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FOREWORD 


Education  currently  consumes  a  vast  amount  of 
resources,  and  it  appears  almost  certain  that  it  will 
consume  much  more  in  the  future.  Since  the  systems  an¬ 
alysis  process  makes  possible  the  evaluation  of  alter¬ 
native  courses  of  action  for  the  achievement  of  instruc¬ 
tional  goals  and  objectives  within  resource  constraints, 
those  in  instructional  management  are  enthusiastic  over 
its  potential  as  a  viable  base  for  decision-making. 
Cost-effectiveness  analysis,  one  aspect  of  systems  an¬ 
alysis,  has  been  shown  to  aid  in  the  better  use  of  re¬ 
sources.  There  are,  however,  a  variety  of  cost-effec¬ 
tiveness  analysis  methods  and  confusion  has  existed  as 
to  the  respective  advantages  and  disadvantages  of  each 
method.  This  guide,  therefore,  is  directed  towards  the 
analysis  of  cost-effectiveness  analysis  itself  from  the 
point  of  view  of  instructional  management.  This  approach 
is  so  that  the  most  appropriate  instructional  cost-ef¬ 
fectiveness  analysis  method  may  be  selected  as  an  aid  to 
decision-making . 
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PREFACE 


There  are  a  variety  of  cost-effectiveness  analy¬ 
sis  methods  and  these  methods  differ  in  approach  and 
precision.  This  guide  will  attempt  to  present  the  most 
applicable  cost-effectiveness  analysis  method  as  a  deci¬ 
sion-assisting  aid  for  instructional  management.  Thus, 
the  approach  taken  is  to:  (1)  explore  what  cost-ef¬ 
fectiveness  methods  are  being  used;  (2)  examine  the 
differences  of  these  methods  in  an  analytical  frame¬ 
work;  and,  (3)  suggest  the  most  appropriate  cost-ef¬ 
fectiveness  analysis  method  for  instructional  manage¬ 
ment  decision-making. 

Due  to  the  complexity  of  the  area  of  systems/ 
cost  analysis,  the  study  will  be  limited  to  the  subject 
of  cost-effectiveness  analysis  itself.  In  addition,  the 
emphasis  will  be  on  the  performance  of  the  cost-effec¬ 
tiveness  analysis  by  someone  other  than  the  instruction¬ 
al  management  decision-maker,  as  a  service  in  support  of 
the  decision-making  process.  The  decision-maker,  in  this 
case,  might  be:  (1)  a  dean  or  department  head  of  an 
academic  institution;  (2)  a  training  manager  or  super¬ 
visor  of  industry  or  government;  or,  (3)  a  superinten¬ 
dent,  principal,  curriculum  director,  or  program  direc¬ 
tor  of  a  school  system. 

Although  the  guide  includes  considerable  detail¬ 
ed  information  involving  the  documentation  of  procedural 
steps  in  various  cost-effectiveness  analytical  techniques, 
it  is  still  less  than  would  be  required  for  an  analyst  to 
use  all  aspects  of  the  procedure.  An  attempt  was  made, 
therefore,  to  limit  the  amount  of  detailed  description 
to  the  minimum  information  requirements  for  the  perfor¬ 
mance  of  a  cost-effectiveness  analysis. 
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ABSTRACT 


The  purpose  of  the  study  is  to  identify  the  differ¬ 
ences  in  cost-effectiveness  analysis  methods  so  that  the  most 
appropriate  one  could  be  selected  as  a  decision-assisting  aid 
for  instructional  management. 

The  principal  phases  of  the  study  are  organized  to  ex¬ 
plore  and  identify  what  cost-effectiveness  methods  are  being 
used;  to  identify  and  examine  the  differences  among  these 
methods  in  an  analytical  framework;  and,  to  determine  the 
criteria  to  be  used  in  selecting  the  most  appropriate  cost- 
effectiveness  method  for  instructional  decision-making. 

The  emphasis  of  the  study  is  on  the  performance  of  cost 
effectiveness  analysis  as  a  service  in  support  of  the  decision 
making  process  by  someone  other  than  the  instructional  manage¬ 
ment.  The  descriptive  detail  provided  in  the  study  is  such 
that  an  analyst  with  moderate  experience  could  perform  the 
indicated  activities. 

The  results  of  study  suggest  that  cost-effectiveness 
analysis  provides  answers  by  a  systematic  process  which  is  re- 
producable  and  accessible  to  critical  examination.  On  the 
other  hand,  cost-effectiveness  analysis  cannot  be  a  substitute 
for  judgment  in  decision-making  by  inductive  reasoning  which 
is  fully  documented.  It  is  believed  that  the  use  of  instruc¬ 
tional  cost-effectiveness  analysis  will  enhance  the  possibil¬ 
ity  that  a  decision  will  result  in  the  selection  of  the  better 
(or  best)  alternative. 
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INTRODUCTION 


Education  the  world  over  is  a  vast  enterprise  and  one 
that  continues  to  grow.  Estimates  vary  concerning  the  pre¬ 
cise  cost  of  education  and  training  in  the  United  States; 

"It's  like  asking  how  many  trees  there  are  in  Russia,"  com¬ 
mented  a  training  program  authority  who  has  attempted  to  deter¬ 
mine  exact  accounting  several  times  (Kleinschrod ,  1967,  p.  18). 
It  is  suggested  that  approximately  64 -million  persons  in  the 
United  States  are  involved  in  some  aspect  of  education  and 
training  and  the  annual  cash  outlay  is  estimated  to  be  an  ex¬ 
penditure  of  $90  billion1  (Goodman,  1973,  overleaf). 

With  the  expenditures  of  resources  of  such  magnitude 
and  with  the  competition  for  these  resources  by  other  needed 
public  enterprise,  educators  must  search  for  ways  to  optimize 
utilization  of  the  instructional 2  dollar.  One  possible  way  to 
attain  maximum  output  from  our  instructional  resources  is  to 
increase  the  effectiveness  of  our  materials,  methods,  facili¬ 
ties,  etc. 


This  purpose  has  little  relation  to  that  once  vaulted 
"cult  of  efficiency"  that  sought  to  bring  business  methods  to 
bear  on  inefficient  schools.  The  purpose  of  sensible  economic 
practices  in  education  has  less  to  do  with  efficiency  than  with 
effectiveness .  It  is  a  question .. .of  turning  out  the  highest- 
quality  product  possible  for  the  funds,  talent,  and  time  expended. 
[U.S.  Congress,  1970,  p.  24). 


Another  way  to  get  more  from  instructional  resources  is 
to  consider  alternative  methods  of  reaching  a  goal  through  sys¬ 
tems  analysis. 

The  systems  approach  is  an  "economic"  approach  to  educa¬ 
tion  in  the  best  sense  of  that  word.  It  provides  a  rational 
method  of  using  a  given  set  of  resources  to  provide  a  system  cap¬ 
able  of  achieving  a  given  set  of  objectives  [Springer,  1S67,  p.  58], 


1 About  50  percent  of  most  municipal  budgets  are  allocated  to 
public  education  (Burnes,  1967,  p.  39)  . 

2 For  the  purpose  of  this  guide,  the  term  "instruction"  will  be 
used  to  refer  to  both  education  and  training. 
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To  some,  systems  analysis  is  characterized  solely  by  auto¬ 
matic  data  processing  techniques  and  is  dismissed  as  being 
modern  gadgetry.  This  may  not  be  true,  for  most  systems  an¬ 
alysis  formulations  are  indistinguishable  from  descriptions 
of  the  scientific  method.  Consequently,  systems  analysis 
roots  reach  back  past  Roger  Bacon  to  Aristotle,  and  not,  as 
some  believe,  only  to  the  RAND  Corporation  and  the  Department 
of  Defense  of  the  1960's  (Oettinger,  1968,  p.  77).  The  newer 
approaches  to  systems  analysis,  however,  attach  more  impor¬ 
tance  to  the:  (1)  establishment  of  quantifiable  objectives 
and  alternatives;  (2)  identification  of  the  alternatives 
cost  and  benefits;  and,  (3)  provision  for  analysis  time 
(McGivney,  1969,  p.  32).  It  must  be  emphasized  that  systems 
analysis  is  most  valuable  as  an  analytic  tool  to  aid  in  in¬ 
structional  management  decision-making,  rather  than  as  a  deci¬ 
sion-maker  itself  (Oettinger,  1968,  p.77). 

Cost-benefit  analysis  and  cost-effectiveness  analysis 
make  up  that  aspect  of  systems  analysis  which  is  concerned 
with  the  selection  of  alternatives  based  on  consideration  of 
the  best  use  of  the  available  resources  to  reach  required  ob¬ 
jectives.  This  analysis  of  resource  usage  requires  synthesiz¬ 
ing  knowledge  about  the  student,  the  instructional  setting,  ob¬ 
jectives,  facilities,  instructional  method,  staff,  standards 
of  achievement  and  other  related  factors.  This  systems/cost- 
eff ectiveness  analysis  approach  has  the  goal  of  establishing 
improved  relationships  between  these  interacting  elements  so 
that  the  total  instructional  enterprise  is  performing  more 
effeciently  and  at  a  higher  level  of  excellence  (Freedman, 

1969,  p.  32) . 

The  need  for  and  value  of  cost-benefit  analysis  and 
cost-effectiveness  analysis  has  been  recognized  in  public  and 
private  education.  It  was  reported  by  the  Commission  on  Educa¬ 
tional  Technology  that  in  improving  learning,  cost  and  cost- 
benefit  relationships  were  the  areas  of  greatest  uncertainty 
(Carpenter,  1970,  p.  5).  A  prominent  private  committee  re¬ 
commended  that : 

School  systems  must  employ  continuously  the  result  of 
cost-benefit  and  cost-effectiveness  analysis  in  order  to  allocate 
effectively  the  resources  available  to  education  and  to  distin¬ 
guish  among  programs  of  high  and  low  priority  [Committee  for 
Economic  Development,  1968,  p.  13], 

In  addition.  President  Nixon  was  quoted  in  the  Washington  Post 
as  saying  that  for  education,  the  1970 's  will  become  tne  "Age 
of  Accountability,"  and  that  one  aspect  will  be  the  judgment  of 
school  programs  relative  to  cost-effectiveness  (Wentworth,  1970)  . 
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It  would  seem  that  the  current  position  on  system/cost- 
effectiveness  analysis  in  support  of  instructional  management 
decision-making  is  that  many  are  enthused  over  the  potentional 
of  it  (McGivney,  1969,  p.  31) --but  who  will  perform  it  and  how 
it  will  be  done  are  more  difficult  considerations.  Systems/ 
cost  analysis  personnel  might  come  from  the  following  areas: 

•  Government.  Those  individuals  as¬ 
sociated  with  the  Department  of 
Defense  and  their  prime  contractors 
who  hope  to  transfer  their  profes¬ 
sional  and  scientific  capabilities 
from  defense  to  education-oriented 
activities  (Lieberman,  1968,  p.  24). 

•  Business.  Independent  consultants 
with  no  vested  interest  in  hardware 
or  a  product  (Herbert,  1967,  p.  45). 

•  Education.  Educators  and  admini¬ 
strators  responsible  for  planning 
(Heinich,  1969,  p.  330). 

Attaining  a  process  by  which  cost-effectiveness  analy¬ 
sis  may  be  applied  to  instructional  decision-making,  however, 
is  a  more  formidable  task.  One  difficulty  is  that  although 
much  has  been  reported  on  the  advantages  of  instructional 
cost-effectiveness,  only  a  limited  number  of  "how  to  do  cost- 
effectiveness"  studies  have  been  published  (Fisher,  1967,  p.  69). 
Another  problem  is  that  the  usefulness  of  cost-effectiveness 
analysis  is  not  fully  understood  by  many  individuals.  This 
lack  of  understanding  is  due  to  the  apparent  coldness  of  the 
tool  as  a  measure  of  decision-making  (Banghart,  1969,  p.  206)  . 

There  are  a  variety  of  cost-effectiveness  analysis  me¬ 
thods  in  use  today  which  differ  in  their  respective  advantages 
and  limitations  (Seiler,  1969,  pp.  1-2).  As  new  approaches 
develop,  these  methods  constantly  undergo  subtle,  but  signifi¬ 
cant,  changes  (Kazanowski,  1968  ,  p.  163)  .  Although  most  of 
these  new  approaches  are  primarily  mathematical  in  nature,  a 
few  are  directed  toward  a  better  treatment  of  problems  that 
cannot  be  handled  by  purely  quantitative  methods  (Quade,  1968, 
p.  242). 

With  the  apparent  lack  of  understanding,  limitations, 
and  wide  differences  of  approaches  in  cost-effectiveness  analy¬ 
sis,  the  following  question  seems  appropriate.  How  can  this 
analytical  tool,  which  was  developed  as  a  decision-assisting 
aid  in  the  selection  of  technical  systems,  be  used  by  instruc¬ 
tional  management  increasing  student  achievement  (Freedman,  1969, 
p.  31)?  The  attempt  to  provide  an  answer  to  this  question  is 
the  purpose  of  this  guide. 


5 


BACKGROUND  OF  COST-EFFECTIVENESS  ANALYSIS 


Cost-effectiveness  analysis  is  not  completely  new.  To 
some  extent,  it  has  always  been  part  of  industrial  and  govern¬ 
ment  planning  and  procurement.  As  recently  as  a  few  years  ago, 
however,  the  use  and  understanding  of  government  cost-effec¬ 
tiveness  analysis  was  inadequate  for  two  major  reasons:  (1) 
all  elements  of  cost  were  not  included  in  the  analysis;  and 
(2)  the  probabilistic  nature  of  system  effectiveness  was  not 
recognized  (Herd,  1965,  pp .  79-80).  By  the  late  1950's  it 
became  obvious  that  technical  improvements  in  governmental 
programs  were  not  always  accompanied  by  improvements  in  total 
program  performance  and  effectiveness.  As  a  consequence,  a 
foundation  for  more  systematic  and  objective  methods  of  mea¬ 
suring  the  effectiveness  of  alternative  courses  of  action  for 
military  systems  was  developed  and  implemented  in  1961  by  the 
Department  of  Defense  (Jakobsberg,  1966,  p.  38).  The  next 
major  step  occurred  when  the  President  directed  the  introduc¬ 
tion  of  an  integrated  planning-programming-budgeting  system 
into  the  Executive  Branch  of  the  Federal  Government;  this 
system  included  cost-effectiveness  analysis  as  a  tool  for  re¬ 
source  allocation  decisions. 

Today,  the  cost-effectiveness  approach  has  been  adopted 
by  many  other  activities  of  government  as  well  as  organizations 
of  private  industry  (Seiler,  1969,  pp.  vii-1) .  According  to 
Jarrett : 


The  significance  of  cost-effectiveness  analysis  today 
stems  from  the  increasing  sophistication  of  our  methods  in  relat¬ 
ing  cost  to  effectiveness  as  compared  with  methods  available  only 
a  few  years  ago.  It  has  been  referred  to  as  "qualified  common 
sense",  which  actually  is  close  to  what  cost-effectiveness  is  all 
about  [Jarrett,  1967,  p.  8]. 


PURPOSE  OF  COST-EFFECTIVENESS  ANALYSIS 

The  purpose  of  cost-effectiveness  analysis  is  to  provide 
the  decision-maker  with  data  on  the  cost  [resource  consumption) 
and  the  probably  effectiveness  (quality  of  results  to  be  a- 
chieved)  for  each  of  several  alternatives,  among  which  he  must 
choose  one  course  of  action.  Cost-effectiveness  analysis  in¬ 
volves  clarifying  the  relationships  between  these  two  factors 


(cost  and  effectiveness)  so  that  the  decision-maker  can  strike 
the  b^t  balance  between  them.  Cost-effectiveness  analysis 
is  not  a  substitute  for  judgment,  experience,  and  common 
sense.  It  is  a  method  for  the  systematic  examination  of  all 
the  subjective  and  objective  data  available  (Heymont,  1965, 

p.  1)  . 


PERFORMANCE  OF  COST-EFFECTIVENESS  ANALYSIS 

The  current  trend  in  the  performance  of  cost-effective¬ 
ness  is  the  application  of  the  method  by  all  levels  of  deci¬ 
sion-making,  in  a  variety  of  functions,  and  for  both  short- 
range  and  long-range  planning  (Heuston  §  Ogawa,  1966,  p.  243). 
The  analysis  itself  may  be  performed  by  the  decision-maker, 
by  a  specialist  within  the  decision-maker's  organization,  or 
the  responsibility  for  cost-effectiveness  analysis  may  be  del¬ 
egated  to  a  specialized  service  organization  (McGivney,  1969, 
p.  31) .  The  latter  approaches  require  close  cooperation  and 
coordination  between  the  cost-effectiveness  analyst  primarily 
responsible  for  the  study,  the  decision-maker  requesting  the 
study,  and  each  of  the  other  individuals  providing  information¬ 
al  inputs  (Fields,  1966,  p.  517).  At  best,  cost-effectiveness 
analysis  functions  as  a  tool  used  in  providing  the  decision¬ 
maker  with  an  analytical  foundation  for  making  sound  objective 
choices  among  the  various  ways  a  problem  might  be  solved  or 
an  objective  met  (Jarrett,  1967,  pp .  7  §  9). 


DEFINITION  OF  COST-EFFECTIVENESS  ANALYSIS 

The  definition  of  cost-effectiveness  analysis  varies, 
for  the  methodology  is  not  precise  enough  to  mean  the  same 
thing  to  all  practitioners  in  the  field  (Seiler,  1969,  p.  1). 
Broadly  defined,  it  is  an  analytic  study  designed  to  assist 
a  decision-maker  in  identifying  a  preferred  choice  among  pos¬ 
sible  alternates  (Quade,  1969,  p.  1).  The  choice  of  alternates 
involves  two  approaches:  (1)  fixed-cost  and  flexible  effec¬ 
tiveness  approach;  and,  (2)  fixed-effectiveness  and  flexi¬ 
ble  cost  approach  (Kazanowski,  1968  ,  pp .  126-127).  Several 
varieties  of  a  more  precise  definition  are  graphically  present¬ 
ed  in  Figure  1  to  illustrate  the  emphasis  different  authori¬ 
ties  have  placed  on  specific  nomenclature. 

Attention  should  also  be  given  as  to  what  is  not  cost- 
effectiveness  analysis.  Kazanowski  calls  this  the  "maximum¬ 
effectiveness  -at-minimum-cost  fallacy"  and  defines  this  mis¬ 
conception  as  follows: 


THE  BASIC  OBJECT  OF  COST-EFFECTIVENESS  ANALYSIS  IS 
TO  ISOLATE  THE  ALTERNATIVE  OR  COMBINATIONS  OF  ALTER¬ 
NATIVES  THAT  GIVE(S): 


1.  The  minimizing  of  or 
costs  subject  to 
some  objective 
constraint 


2.  The  greatest  or 

objective  effect¬ 
iveness  for  a  given 
expected  cost 

3.  The  greatest  or 

effectiveness  for 

any  given  cost 


4.  The  most  productive  or 
alternative  if  the 
costs  are  equal 


a  maximizing  of  some 
physical  measure  of 
the  objective  out¬ 
put  subject  to  cost 
constraint  (Quade, 

1969,  p.  2). 

a  given  expected  object¬ 
iveness  for  the 
least  expected  cost 
(Seiler,  1969,  p.  1). 

a  required  or  chosen 
degree  of  effective¬ 
ness  for  the  least 
cost  (McGivney  and 
Nelson,  1969,  p.  105). 

the  least  costly 
alternative  that 
produces  equal 
benef i ts . 


If  the  alternatives  are  of  unequal  costs  and  produce 
unequal  benefits,  cost-effectiveness  analysis  will 
provide  information  for  comparing  costs  and  benefits 
(Office  of  the  Assistant  Secretary  of  Defense,  1971 
P-  5). 


Figure  1.  Variation  in  Cost-Effectiveness 
Analysis  Definition 
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Many  references  can  be  found  in  which  the  statement  is 
made  directly  or  indirectly  that  cost-effectiveness  is  a  tech¬ 
nique  by  which  one  determines  that  alternative  which  yields  the 
maximum  effectiveness  as  a  minimum  cost.  Or,  in  other  words, 
that  alternative  is  sought  by  which  the  effectiveness  is  max¬ 
imized,  while  the  cost  is  minimized.  In  reality  the  attempt  to 
find  such  an  alternative  is  doomed  to  failure  at  the  onset  be¬ 
cause  such  an  alternative  does  not  exist.  Hitch  and  McKean  state: 
"Actually,  of  course,  it  is  impossible  to  choose  that  policy  which 
simultaneously  maximizes  gain  and  minimizes  cost,  because  there  is 
no  such  policy."  [Kazanowski,  1968,  p.  160]. 


DIFFERENCES  BETWEEN  COST-EFFECTIVENESS 
AND  COST-BENEFIT  ANALYSIS 

Confusion  also  exists  concerning  the  difference  bet¬ 
ween  cost-effectiveness  analysis  and  cost-benefit  analysis 
(Fisher,  1967,  p.  66).  The  principal  differences  in  these 
two  methods  are  the  elements  of  output  and  of  time -phasing . 


Output 


The  term  "cost-effectiveness"  usually  applies  to  situa 
tions  in  which  it  is  difficult  to  quantify  the  alternate  out¬ 
puts  in  terms  of  dollar  values.  Cost-benefit  usually  consi¬ 
ders  only  the  monetary  worth  of  outcomes. 


Time  Phasing 

The  term  "cost-effectiveness"  is  usually  used  to  select 
the  most  appropriate  choice  from  the  predicted  alternative 
outcomes;  the  term  "cost-benefit"  is  most  often  applied  in 
selecting  the  most  appropriate  choice  from  the  calculated  pre¬ 
sent  (McGivney  and  Nelson,  1969,  p.  105). 

Cost-effectiveness  analysis  is  specifically  directed  to 
problems  in  which  the  output  cannot  be  evaluated  in  market  prices, 
but  where  the  inputs  are  substitutable  at  exchange  relationships 
developed  in  the  market  [Niskanen,  1969,  p.  18]. 

SUBJECT  APPROPRIATENESS  FOR  COST-EFFECTIVENESS  ANALYSIS 

Certain  prerequisites  should  be  met  before  a  subject 
is  appropriate  for  cost-effectiveness  analysis  (Kazanowski, 
1968,  p.  114).  The  primary  appropriate  prerequisite  is  that 
the  subject  should  have  two  characteristics:  first,  and  most 
important,  the  subject  must  be  relevant;  second,  but  less 


k 


1 


important,  it  should  be  capable  of  being  measured  both  in  re¬ 
quired  resources  and  outcome.  These  two  characteristics  often 
appear  conflicting.  The  most  relevant  phenomena  or  events  are 
often  very  difficult  to  measure,  the  least  relevant  are  usually 
easy  to  measure  (Niskanen,  1969,  p.  20).  Cost-effectiveness 
analysis  should  be  performed  only  when  the  following  appro¬ 
priate  prerequisites  are  met: 

•  The  correct  problem  must  be  recog¬ 
nized  . 

•  Realistic  objectives  that  bear  a 
functional  relationship  to  the  prob¬ 
lem  must  be  determined  (Jakobsberg, 

1966 ,  p .  38) . 

•  Common  goals,  objectives,  or  mis¬ 
sions  of  the  alternative  must  be 
identifiable  and  at  least  theoret¬ 
ically  attainable. 

•  Constraints  for  bounding  the  problem 
must  be  discernible. 


•  Alternative  means  of  meeting  the 
goals  must  exist  (Kazanowski,  1968, 
p.  114)  . 

The  meeting  of  these  requirements  is  not  always  simple; 
consequently,  if  there  is  no  freedom  of  choice  in  problem  solv¬ 
ing,  cost-effectiveness  analysis  is  not  appropriate  (Jarrett, 
1967,  p.  8) . 


RELATIONSHIP  OF  COST-EFFECTIVENESS  TO  SYSTEMS  ANALYSIS 

In  order  to  place  cost-effectiveness  in  proper  perspec¬ 
tive,  it  is  first  necessary  to  place  it  in  relation  to  the  sys¬ 
tems  analysis  process.  Cost-effectiveness  analysis  is  not  a 
method  for  measuring  either  cost  or  effectiveness  after  the 
fact,  for  its  modern  applications  are  concerned  with  the  pre¬ 
diction  and  evaluation  of  an  alternative's  worth  (English,  1968, 
p.  7).  The  term  "systems  analysis"  is  used  to  distinguish  this 
broader  analysis  from  a  narrow  cost-effectiveness  aspect  of 
analysis.  Cost-effectiveness  analysis  is  that  aspect  of  sys¬ 
tems  analysis  which  emphasizes  the  systematic  investigation  of 
the  decision-maker's  alternatives  relative  to  objective  criter¬ 
ia,  costs,  and  effectiveness  comparison,  in  addition  to  the 
other  trade-off  factors  associated  with  alternative  selection 
(Quade,  1969,  p.  3).  Figure  2  graphically  illustrates  this 
emphasis . 


BROADLY  DEFINED 
PHASES  OF 
SYSTEMS  ANALYSIS 


GOAL/OBJECTIVES/ 

REQUIREMENTS 

DEFINITION 


VARIOUS  AIDS  IN  THE 
DECISION-MAKING 
PHASE 


ALTERNATIVE 

RELIABILITY 

ANALYSIS  AND 

DEVELOPMENT 

OTHER  TECHNIQUES 

i 

ALTERNATIVE 

TRADE-OFF 

rr\  ct  rccrrm/rwccc 

wUj 1  L  r I  L  L 1  1 V  L  N  L  j  j 

ANALYSIS 

STUDIES 

r 

i 

1 

ALTERNATIVE 

_ i 

CONSTRAINT 
IDENTIFICATION  AND 

SELECTION 

OTHER  TECHNIQUES 

Note:  After  Heaton,  1969,  p.  35 


Figure  2.  Relationship  of  Cost  Effectiveness 
to  Systems  Analysis 
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It  can  be  seen,  then,  that  the  difference  between  cost- 
effectiveness  analysis  and  systems  analysis  is  a  matter  of  emphasis. 
If  the  emphasis  is  on  finding  significant  differences  in  the  costs 
or  resource  requirements  among  the  available  alternatives  for 
carrying  out  some  specified  task,  the  analysis  is  generally  re¬ 
ferred  to  as  a  cost-effectiveness  analysis.  The  systems  analyst, 
on  the  other  hand,  is  likely  to  be  forced  to  deal  with  problems 
in  which  the  difficulty  is  deciding  what  ought  to  be  done,  not 
simply  how  to  do  it.  System  analysis  thus  puts  greater  emphasis 
on  the  suitability  of  the  task  and  the  augmentation  of  alternatives. 
In  both  system-analysis  and  cost-effectiveness  work  it  is  impor¬ 
tant  to  note,  however,  that  after  an  appropriate  evaluation  one 
will  be  in  a  much  better  position  to  make  a  decision,  even  though 
he  may  not  find  the  decision  easier  to  make  [Bell,  1964,  pp.  1-2], 


CATEGORIES  OF  COST-EFFECTIVENESS  ANALYSIS 

The  confusion  over  the  role  cost-effectiveness  analysis 
plays  in  the  systems  analysis  process  seems  to  arise  from  the 
fact  that  some  practitioners  perceive  cost-effectiveness  analy¬ 
sis  to  be  the  systems  analysis  process  itself;  whereas,  others 
see  the  process  of  cost-effectiveness  as  being  related  to  and 
a  sub-set  of  the  total  systems  design  process.  As  noted  above, 
both  interpretations  of  the  role  of  cost-effectiveness  analy¬ 
sis  could  be  appropriate,  depending  upon  the  time  and  condi¬ 
tions  that  the  analysis  requires.  There  are,  however,  three 
categories  of  cost-effectiveness  analysis  which,  in  practice, 
are  not  clearly  distinct.  These  categories  are:  (1)  system/ 
component  configuration  study;  (2)  system  comparison  study; 
and,  (3)  suprasystem  comparison  study  (Heuston  5  Ogawa,  1966, 
p.  245).  The  relationship  of  these  categories  is  exhibited  in 
Figure  3  and  a  detailed  description  follows. 


System  Configuration/Component  Study 

In  this  category  the  emphasis  is  on  the  selection  of  the 
particular  configuration,  or  characteristics,  of  a  single  sys¬ 
tem  with  different  components.  For  example,  a  closed-circuit 
instructional  television  system  versus  an  open-circuit  instruc¬ 
tional  television  system.  This  category  of  study  is  probably 
the  one  most  often  performed. 

Cost  Treatment.  Typically,  system/component  configura¬ 
tion  studies  have  the  following  characteristics  relating  to  the 
treatment  of  costs: 

•  Detailed  costs  are  obtained  on  all 
components . 
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•  Various  mathematical  models  are 
used  to  simulate  and  optimize  dif¬ 
ferent  possible  configurations  of 
the  components .  The  results  yield 
the  predicted  overall  costs  of  the 
various  alternatives  under  study. 

•  Costs  are  compiled  by  summing  the 
yearly  costs  over  the  life  cycle 
of  the  systems  rather  than  by  com¬ 
piling  time-phased  costs. 

Effectiveness  Treatment.  As  in  the  treatment  of  cost, 
the  effectiveness  measure  used  in  the  system  configuration/com¬ 
ponent  study  will  be  detailed  in  terms  of  capabilities  and  per¬ 
formance.  In  most  cases,  effectiveness  will  not  be  a  simple 
overall  measure  which  ultimately  would  determine  alternative 
selection  (Hatry,  1969,  pp .  45,  47-48). 

It  should  be  noted  that  trade-off  analysis  at  the  detail 
level  of  selecting  subcomponents  is  classified  as  value  analy¬ 
sis  rather  than  cost-effectiveness  analysis  (Kazanowski,  1968, 
p.  153).  An  example  of  this  type  of  analysis  would  be  the  se¬ 
lection  of  the  least  expensive  instructional  television  receiv¬ 
er  that  complies  with  given  specifications. 


System  Comparison  Study 

Here,  the  emphasis  is  on  comparing  two  or  more  systems 
for  the  same  objective.  The  emphasis  is  on  intersystem,  not 
intrasystem,  analysis.  It  is  presumed  that  each  competing  sys¬ 
tem  has  already  been  sub-optimized  as  to  its  configuration, 
through  the  system/configuration  component  study  discussed  pre¬ 
viously.  An  example  of  this  type  of  study  would  be  to  compare 
the  merits  of  a  programmed-instructional  system  versus  an  auto¬ 
tutorial  system  (where  each  has  been  instructionally  validated) 

Cost  Treatment.  Typically,  the  system  comparison  study 
has  the  following  characteristics  relating  to  system  costs: 

•  Costs  are  generally  required  in 
less  detail  than  in  system/compo- 
ment  configuration  studies  where 
the  emphasis  was  on  components  of 
competing  similar  characteristics. 

•  The  specific  spread  of  costs  over 
the  life  cycle  of  the  system  is 
usually  ignored,  or  is  treated  as 
a  secondary  problem. 
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Effectiveness  Treatment.  The  effectiveness  measure  is 
basically  quite  similar  to  that  used  in  system/component  con¬ 
figuration  studies. 


Suprasystem  Comparison  Study 

This  third  cost-effectiveness  study  category  is  less 
frequently  performed  as  a  strictly  cost-effectiveness  activity 
Activities  at  this  level  of  analysis  are  usually  described  as 
cost-benefit.  The  problem  here  is  to  assess  the  merits  of  al¬ 
ternatives  of  widely  differing  capabilities  and  cost  elements. 
An  example  would  be  to  analyze  the  cost-effectiveness  of  com¬ 
peting  instructional  systems  such  as  institutional  study  as 
opposed  to  on-the-job  training. 

Cost  Treatment.  In  suprasystem  comparison  studies, 
costs  are  each  time-phased  to  the  extent  possible,  to  provide 
estimates  of  annual  resource  requirements  (Hatry,  1969,  pp .  48 
§50). 

Effectiveness  Treatment.  The  determination  of  a  mea¬ 
surement  of  effectiveness  is  especially  difficult  for  those 
cases  where  the  systems  are  experimental,  the  objectives  are 
of  unequal  priority,  and  the  output  varies  in  type  and  relia¬ 
bility  (Heuston  §  Ogawa,  1966,  p.  245)  . 

The  problem  of  the  analysis  of  differing  costs  and  capa 
bilities  of  alternatives  requires  the  development  of  some  mea¬ 
sure  of  structure  for  comparing  these  alternatives  on  a  some¬ 
what  equal  basis  (Department  of  Defense,  1971,  p.  5). 


PROCESSES  OF  COST-EFFECTIVENESS  ANALYSIS 

In  addition  to  the  confusion  from  the  differences  of 
cost-effectiveness  analysis  categories,  differences  in  the  pro 
cess  of  the  analysis  itself  has  compounded  the  lack  of  under¬ 
standing.  Yet,  in  all  the  variations,  two  processes  are  per¬ 
formed:  (1)  the  procedural  process;  and,  (2)  the  applica¬ 

tion  process . 


Procedural  Process 

This  cost-effectiveness  analysis  process  can  be  defined 
as  the  documentation  or  writing  of  instructions  on  how  to  per¬ 
form  the  analysis  in  a  step  procedural  sense.  This  type  of 
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process  is  also  known  as  model  building;  i.e.,  a  conceptual 
mock-up  or  detailed  paper-pencil  description  of  the  actual 
analysis  to  be  performed  at  some  later  time. 


Application  Process 

The  second  cost-effectiveness  analysis  process  refers 
to  the  application  of  the  procedural  process  by  applying  a 
particular  model  to  a  real-world  practical  situation  (Speagle, 
1969,  p.  22) . 
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STANDARD  PROCEDURE  FOR  COST-EFFECTIVENESS  ANALYSIS 


The  chief  value  of  cost-effectiveness  analysis  is  its 
broad  ability  to  provide  a  general  framework  to  assist  in  the 
solution  of  problems  in  a  wide  range  of  topical  areas.  Cost- 
effectiveness  analysis  differs  from  many  other  decision-assist¬ 
ing  tools  which  are  limited  to  accepting  only  certain  classes 
of  problems. 

The  very  fact  that  cost-effectiveness  analysis  can  as¬ 
sume  characteristics  dictated  by  a  unique  objective  and  res¬ 
pond  to  the  category,  level,  and  process  of  analysis  as  re¬ 
quired  provides  a  lack  of  uniformity  in  the  analysis  itself 
(Jakobsberg,  1966,  p.  37).  In  addition,  the  apparent  lack  of 
uniformity  in  most  examples  of  cost-effectiveness  analysis, 
and  the  lack  of  consistent  documentation,  has  led  some  persons 
to  question  the  merits  of  this  type  of  analysis  as  an  aid  to 
decision-making  (Kazanowski,  1968,  p.  113). 

On  the  other  hand,  certain  authors  and  authorities  in 
the  field  indicate  that  a  complete  "how- to-do - it"  document,  or 
a  "cookbook,"  would  be  impossible  to  prepare  (Jakobsberg,  1966, 
p.  37).  This  may  be  because,  at  the  current  stage  of  develop¬ 
ment,  cost-effectiveness  analysis  is  more  an  art  than  a  science 
(Fisher,  1967,  p.  70).  Nevertheless,  it  is  suggested  that  the 
majority  of  approaches  to  cost-effectiveness  analysis  have  cer¬ 
tain  common  functional  elements  which  may  be  synthesized  into 
a  standard  procedural  approach.  These  steps  are  illustrated 
in  Figure  4  and  include  the  following  inputs  from  others,  anal¬ 
ysis,  and  outputs: 

•  STEP  1:  Review  definition  of  the  desired  gen¬ 
eral  goals  and  specific  objectives  that  the  de¬ 
sign  is  to  meet  or  fulfill. 

•  STEP  2:  Review  identification  of  the  parameters 
or  requirements  essential  for  the  attainment  of 
the  desired  goals/objectives. 

•  STEP  3:  Review  development  of  the  alternatives 
for  accomplishing  the  goals/objectives/require¬ 
ments. 

•  STEP  4:  Establish  an  evaluation  criteria  (mea¬ 
sures),  for  cost  and  effectiveness,  that  relate 
alternative  capabilities  to  the  goals/objectives/ 
requirements. 
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INPUT 


STEP  1 

REVIEW  DEFINITION 
OF  GENERAL  GOALS/ 
SPECIFIC  OBJECTIVE! 


STEP  7 

SELECT  FIXED- 
COST  OR  FIXED¬ 
EFFECTIVENESS 
APPROACH 


STEP  2 

REVIEW  IDENTIFICA¬ 
TION  OF  PARAMETERS 
OR  REQUIREMENTS 


STEP  8 

CREATE  ALTERNATIVE 
VERSUS  CRITERIAN 
ARRAY 


STEP  3 

REVIEW  DEVELOPMENT 
OF  ALTERNATIVES 


STEP  9 

ANALYZE  MERITS 
OF  ALTERNATIVES 


STEP  4 

ESTABLISH  COST 
AND  EFFECTIVENESS 
EVALUATION 
CRITERIA 


STEP  10 

ANALYZE  UNCER¬ 
TAINTY  FACTORS 


STEP  5 

DETERMINE  COST 
OF  ALTERNATIVES 


STEP  11 

DOCUMENT  BASIS  OF 
PREVIOUS  STEPS; 
SUBMIT  FINDINGS 


STEP  6 

DETERMINE  CAPA¬ 
BILITIES  OR 
EFFECTIVENESS  OF 
ALTERNATIVES 


OUTPUT 


Note:  After  Pearson,  1972, 


Figure  4.  Cost-Effectiveness  Analysis  Standard  Procedure 
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•  STEP  5:  Determine  the  cost  of  alternatives 
in  terms  of  evaluation  criteria. 

•  STEP  6:  Determine  the  capabilities  or  effec¬ 
tiveness  of  the  alternatives  in  terms  of  eval¬ 
uation  criteria. 

•  STEP  7 :  Select  fixed-cost  or  fixed-effective¬ 
ness  approach. 

•  STEP  8:  Create  an  alternatives  versus  criter- 
ian  array. 

•  STEP  9:  Analyze  the  merits  of  alternatives. 

•  STEP  10:  Analyze  the  uncertainty  factors. 

•  STEP  11:  Document  the  rationale,  assumptions, 
and  analyses  underlying  the  previous  steps  and 
submit  findings  (Pearson,  1973,  p.  34). 

The  following  discussion  of  these  procedures  suggests  guide¬ 
lines  to  be  followed  for  a  standardized  approach  to  the  analy¬ 
sis  of  system/cost-effectiveness.  Although  the  presented 
steps  are  in  an  order  in  which  they  would  usually  be  performed, 
changes  in  the  sequence  are  acceptable  and  should  fit  the  need 
of  the  subject  under  evaluation  (Kazanowski,  1968,  pp.  115- 
116)  . 


STEP  i:  REVIEW  DEFINITION  OF  THE  GOALS  AND  OBJECTIVES 

Before  cost-effectiveness  analysis  can  be  considered, 
the  problem  must  be  identified  and  the  goals/objectives  must 
be  defined,  at  least  in  a  general  nature.  Obvious  as  this 
may  seem,  the  perception  of  the  problem  that  needs  solving, 
and  the  establishment  of  goals  and  objectives,  are  very  often 
complex  and  involved  tasks  for  which  there  are  no  techniques 
and  tools  besides  "logical  thinking"  (Jakobsberg,  1966,  pp .  37- 
38).  Without  such  an  identification  of  goals/objectives  there 
is  no  framework  for  structuring  the  subsequent  evaluations. 

The  following  are  several  points  to  consider  in  reviewing  the 
general  goals/objectives  definition. 

If  goals/objectives  specified  are  in  too  general  terms, 
the  constraints  established  for  bounding  the  evaluation  are 
often  only  the  product  of  the  analyst.  In  addition,  care  must 
be  exercised  not  to  identify  the  goals/objectives  in  such  a 
manner  as  to  bias  the  evaluation  by  including  requirements  of 
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such  specific  nature  that  they  exclude  potential  candidate 
alternatives  from  consideration. 

On  the  other  hand,  care  must  also  be  taken  not  to  make 
goals/objectives  too  specific  or  they  will  limit  the  scope  of 
possible  candidate  alternatives  by  implicitly  defining  alter¬ 
native  concepts  rather  than  just  the  desired  goals/objectives 
A  potential  danger  always  exists  in  that  the  goals/objectives 
originator  may  specify  a  goal/objective  that  is  unattainable 
by  means  of  current  technology. 


STEP  2:  REVIEW  THE  IDENTIFICATION  OF  THE  PARAMETERS  OR 
REQUIREMENTS 

The  basic  purpose  of  identifying  and  defining  the  gen¬ 
eral  goals/objectives  is  to  aid  in  the  identification  of  re¬ 
quirements  essential  to  attaining  the  defined  goals/objectives. 
The  goals/objectives  and  requirements  should  be  identified  and 
specified  within  the  required  parameters.  This  specification 
is  used  to  further  reduce  the  possibility  of  biasing  the  cost- 
effectiveness  analysis.  The  confusion  brought  about  by  close 
relationship  between  goals/objectives  and  requirements  is  fur¬ 
ther  compounded  by  the  variety  of  categories  of  cost-effective¬ 
ness  studies  (i.e.,  suprasystem  comparison  study  and  system 
configuration/component  study) .  The  need  for  detailed  speci¬ 
ficity  can  be  illustrated  by  the  following  simplified  example 
of  the  general  goal/objective  of  eliminating  poverty  of  migrant 
agricultural  workers  by  vocational  skill  training.  One  re¬ 
quirement  would  be  the  identification  of  current  labor  market 
vocations  needed  so  that  detail  job  performance  training  spe¬ 
cifications  could  be  formulted.  The  resulting  requirement  spe¬ 
cification  might  detail  tool  or  machine  operator  competencies 
needed,  as  well  as  conditions  and  level  of  performance  expect¬ 
ed  after  training  completion.  The  major  factors  for  consi¬ 
deration  when  reviewing  the  identification  of  goals/objectives 
and  requirements,  and  the  development  of  specifications  are 
as  follows, 

•  Verify  the  establishment  of  a  relationship  that 
converts  goals/objectives  into  some  unit  of  mea¬ 
sure  for  both  cost  and  effectiveness.  For  ex¬ 
ample,  number  completions  at  a  given  level  of 
achievement. 

•  Be  alert  for  errors  of  commission,  which  are 
just  as  important  as  errors  of  omission.  If 
goals/objectives  that  are  not  necessarily  es¬ 
sential  are  identified  as  requirements,  they 
can  strongly  prejudice  the  subsequent  cost-ef¬ 
fectiveness  evaluation  (Kazanowski,  1968,  pp . 

117-119) . 
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STEP  3:  REVIEW  DEVELOPMENT  OF  THE  ALTERNATIVES 

As  in  defining  the  goals/objectives  and  identifying  the 
requirements,  developing  valid  alternative  means  or  courses  of 
action  that  can  satisfy  the  goals/objectives  is  a  mental  exer¬ 
cise  for  which  no  current  scientific  technique  exists  (Jakobs- 
berg ,  1966,  p.  39).  This  activity  is  to  create  two  or  more 
alternative  concepts  of  ways  to  achieve  the  goals/objectives 
within  the  defined  parameters  or  specifications.  Frequently, 
considerable  imagination  and  creativity  are  required  to  de¬ 
velop  effective  competitive  candidate  alternatives.  The  re¬ 
sults  of  cost-effectiveness  analysis  can  be  no  better  than  the 
conception  and  development  of  attractive  candidate  alternatives 
(Kazanowski,  1968,  p.  120). 

There  are  several  problems  associated  with  the  review 
of  the  alternative  development.  The  most  serious  problem  or 
source  of  defects  is  attention  bias.  This  is  frequently 
caused  by  a  cherished  belief  or  an  unconscious  adherence  to 
what  we  might  call  a  "party  line."  This  attention  bias  may 
result  in  the  unwarranted  favoring  of  a  particular  alternative 
by  the  analyst  (Quade,  1969,  p.  8)  .  Another  problem  is  the 
degree  of  detail  associated  with  available  information  on  al¬ 
ternatives.  Since  the  purpose  of  cost-effectiveness  analysis 
is  to  aid  the  decision-maker  in  deciding  which  alternative 
should  be  selected,  specific  details  on  possible  alternatives 
are  generally  lacking.  Too  little  detail  usually  results  in 
a  large  variance  in  estimated  alternative  effectiveness  and 
cost.  On  the  other  hand,  to  require  that  the  candidate  al¬ 
ternative  be  designed  in  detail,  before  being  evaluated,  would 
defeat  the  basic  purpose  and  value  of  cost-effectiveness  an¬ 
alysis.  In  addition,  the  development  of  new  and  novel  alter¬ 
natives  might  be  discouraged  by  detail  design  requirements. 

This  factor  would  tend  to  favor  existing  or  more  traditional 
alternatives.  The  balance  between  inadequate  detail  and  ex¬ 
cessively  rigorous  specificity  should  be  maintained. 


STEP  4:  ESTABLISH  COST  AND  EFFECTIVENESS  EVALUATION  CRITERIA 

The  criteria  specifies:  (1)  the  measure  of  cost;  (3) 
the  measure  of  effectiveness;  and,  (2)  the  relationship  bet¬ 
ween  them.  All  will  be  used  in  the  selection  of  one  alternative 
from  the  several  alternatives.  The  measure  of  cost  must  be 
consistent  with  tie  overall  framework  of  the  resource  alloca¬ 
tion  problem;  e.g.,  net  future  total  cost  of  ownership  for 
a  certain  number  of  years.  The  measure  of  effectiveness 
should  express  the  extent  to  which  the  goals/objectives  and 
requirements  are  being  accomplished.  The  relationship  between 
the  cost  and  effectiveness  must  be  expressed  in  functional 
terms;  e.g.,  in  a  mathematic  equation,  graphic  display,  or 
model  (Bryk,  1965,  p.  1). 
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The  selection  of  appropriate  and  adequate  cost  and  ef¬ 
fectiveness  evaluation  criteria  is  based  on  judgment  and  ex¬ 
perience.  The  omission  of  significant  criteria  could  readily 
invalidate  the  results  of  a  cost-effectiveness  analysis.  The 
inclusion  of  numerous  criteria  to  be  on  the  safe  side,  how¬ 
ever,  can  result  in  mental  paralysis  when  the  optimum  alter¬ 
native  selection  is  being  made.  The  greater  the  number  of 
criteria,  the  greater  the  probability  that  even  unlikely  can¬ 
didates  will  excel  in  some  criterion,  making  alternative  sys¬ 
tem  selection  difficult.  One  simple  test  of  the  adequacy  or 
completeness  of  cost  and  effectiveness  evaluation  criteria  is 
to  question  whether  one  alternative  could  excel  in  most  of 
the  criteria  established  and  still  not  be  selected  as  the  pre¬ 
ferred  choice.  If  the  answer  is  affirmative,  important  cri¬ 
teria  could  be  missing.  Considerable  insight  into  the  sub¬ 
tleties  of  the  goals/objectives  and  requirements  is  usually 
necessary  for  the  establishment  of  meaningful  alternative 
evaluation  criteria  (Kazanowski,  1068,  p.  123).  Certain  eval¬ 
uation  criteria  of  cost  and  effectiveness  suggested  by  a  var¬ 
iety  of  authors  are  shown  in  Figure  5. 


Develop  Measurers  of  Cost 

The  choice  of  a  particular  alternative  for  accomplish¬ 
ing  the  goals/objectives  implies  that  certain  specific  re¬ 
sources  will  be  consumed  and  could  no  longer  be  used  for  other 
purposes.  These  are  the  costs.  In  future  time  period  analy¬ 
sis  most  costs  could  be  indicated  as  money  expenditures,  but 
their  true  measure  should  be  expressed  in  terms  of  the  values 
of  alternative  opportunities  which  are  precluded  because  of 
this  expenditure  (Quade,  1969,  p.  4).  One  of  the  most  diffi¬ 
cult  problems  in  establishing  cost  evaluation  criteria  is  the 
determination  of  appropriate  cost  measures.  The  choice  of  the 
proper  cost  measure  involves  both  difficulties  and  controversy 
within  the  broad  categories  of  time-phased  analysis  (cost  dis¬ 
tributions  over  extended  time  periods)  and  non- time-phased 
analysis . 


Cost  Measures  for  Non-time-phased  Analysis.  For  the 
purpose  of  non- time-phased  analysis  (such  as  applied  to  the 
system/component  level  configuration  or  system  level  compari¬ 
son  study)  the  following  three  cost  measures  are  commonly  sug¬ 
gested. 


•  Initial  cost  (including  research  and  develop¬ 
ment  and  initial  investment,  to  achieve  the  as¬ 
sumed  goals/objectives)  plus  X  years  of  operat¬ 
ing  costs.  The  choice  of  what  X  should  be  is  a 
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MEASURES  OF  EFFECTIVENESS 


I.  STUDENT/GROUP-ORIENTED 

II.  SCHOOL-ORIENTED 

III.  COMMUNITY  ORIENTED 


A.  UTILITY 

B.  PRODUCTIVITY 

C.  WORTH 

D.  MERIT 

E.  BENEFIT 

F.  GAIN 

G.  VALUE  RECEIVED 


1 . 

PERFORMANCE 

1 1  . 

REPAIRABILITY 

2. 

ECONOMY 

12. 

GROWTH  POTENTIAL 

3. 

SAFETY 

13. 

DEPENDABILITY 

4. 

AVAILABILITY 

14. 

CAPABILITY 

5. 

FLEXIBILITY 

15. 

TECHNICAL  CONFI- 

6. 

PRESTIGE 

DENCE 

7. 

MAINTAINABILITY 

16. 

INFORMATION  YIELD 

8. 

RELIABILITY 

17. 

VERSATILITY 

9. 

PROBABILITY  OF 

18. 

SPILLOVER 

SUCCESS 

EFFECTS 

10 

.EVOLUTIONARY 

19. 

TECHNICAL 

DEVELOPMENT 

DESIRABILITY 

Note:  After  Pearson,  1972,  p.  42 
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problem  but  is  is  most  commonly  taken  as  five 
and  occasionally  ten  years. 

•  Adjusted  initial  cost  plus  X  years  of  operat¬ 
ing  costs.  The  difference  here  is  only  the 
use  of  adjusted  initial  cost  in  which  case  the 
costs  are  adjusted  to  take  into  consideration 
the  estimated  residual  values  for  each  alter¬ 
native.  The  residual-value  problem  occurs  most 
frequently  when  comparing  choices  between  al¬ 
ternatives  which  either:  (1)  could  be  used 
for  additional  goals/objectives;  or,  (2)  the 
life-cycle  of  alternatives  differs  widely. 

The  latter  situation  is  the  most  often  cause 
for  initial  cost  adjustment. 

•  The  above  first  or  second  cost  measure,  but 
discounted  in  some  manner.  Discounting  has 
been  used  in  order  to  perform  two  functions : 

(1)  as  an  adjustment  for  the  increasing  time 
period  uncertainties;  and,  (2)  to  indicate 
economic  impact. 


Cost  Measures  for  Time-Phased  Analysis.  For  the  supra- 
system  comparison  studies,  there  are  three  general  cost  mea¬ 
sures  suggested  to  be  used  in  examining  each  alternative. 

These  are: 

•  Annual  funding/budget  requirements. 

•  Cumulative  funding/budget  requirements . 

•  Present  expenditure  worth  at  a  selected  dis¬ 
count  rate. 

The  problem  in  all  of  the  above  measures  is  to  determine  how 
far  out  the  study  should  be  carried  in  order  to  consider  all 
time  periods  pertinent  to  the  purpose  of  the  study.  At  some 
point  the  effect  of  the  subsequent  years  would  have  negligible 
effect  upon  today’s  decisions. 

There  is  probably  no  ideal  answer  to  the  question  of 
the  cost  measure  to  be  used,  in  either  time-phased  or  non¬ 
time-phased  analysis.  No  single  cost  measure  incorporates 
all  pertinent  elements  and  it  may  well  be  desirable  to  use 
more  than  one  measure  to  give  the  proper  cost  perspective 
(Hatry,  1967,  pp.  64-67)  . 
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Develop  Measurers  of  Effectiveness 

Of  all  the  components  or  constituents  of  cost-effec¬ 
tiveness  analysis  the  cost -measure  concept  has  found  the 
least  acceptance  as  a  valid  measurement  criterion.  Less  dif¬ 
ficult  was  the  acceptance  of  the  effectiveness -measure  cri¬ 
terion  CBreckner,  1967,  p.  83).  Proper  effectiveness  criteria 
selection,  however,  is  one  of  the  most  formidable  problems  of 
cost-effectiveness  analysis.  If  the  goals/objectives  are  not 
trivial,  then  real  world  facets  are  generally  complex  and  have 
widespread  implications;  consequently  the  effectiveness  cri¬ 
teria  may  be  broad  and  numerous.  If  the  scope  of  the  goals/ 
objectives  is  reduced,  usually  so  is  the  number  of  significant 
criteria.  Even  if  the  scope  of  the  problem  is  significantly 
reduced,  it  is  virtually  impossible  to  reduce  the  total  cost- 
effectiveness  analysis  to  a  single  easily  evaluated  effective¬ 
ness  criterion  (Kazanowski,  1968,  p.  152).  On  the  other  hand, 
the  more  narrow  and  fewer  in  number  the  measure  of  effective¬ 
ness  criteria  become,  the  more  limited  will  be  the  eligible 
alternatives  to  be  considered.  It  is  often  desirable,  there¬ 
fore,  to  broaden  a  cost-effectiveness  analysis  study  to  in¬ 
clude  more  effectiveness  criteria  in  order  to  insure  a  wider 
range  of  alternatives  (Breckner,  1967,  p.  58).  Well  defined 
and  explicit  measures  of  effectiveness  are  difficult  to  devel¬ 
op  and  rare.  Typically,  these  measures  are  replaced  by  two 
devices:  (1)  performance  identifiers;  and  (2)  standards. 


Performance  Identifiers.  These  effectiveness  measures 
relate  to  the  administrative  control  of  functions  where  output 
is  inadequately  defined.  Usually,  performance  measures  are 
specified  on  an  activity  basis  to  give  an  estimate  of  work 
performed.  The  performance  identifier  tends  to  be  a  device  to 
measure  work  performance  and  is  designed  to  detect  the  varia¬ 
tions  in  work  performance  levels.  In  fact,  however,  perfor¬ 
mances  may  have  little  relation  to  the  final  output. 


Standards.  These  effectiveness  measures  are  much  broad¬ 
er  than  performance  identifiers.  Government  procurement  ac¬ 
tivities  are  characterized  by  the  use  of  standards,  which  may 
be  of  two  types:  technical  performance  standards,  and  gener¬ 
alized  standards.  Considerable  overlap  exists,  however,  and 
many  problems  are  common  to  both. 

•  Technical  performance  standards  exist  through¬ 
out  government  and  industry  and  are  needed  to 
give  a  vital  measure  of  uniformity  and  inter¬ 
changeability  of  parts,  components,  assemblies, 
and  processes.  For  example,  both  the  govern¬ 
ment  Bureau  of  Standards  and  the  industrial 


Underwriters  Laboratories  develop  operational 
and  performance  standards  for  electrical  items. 

These  standards  are  observed  by  those  manu¬ 
facturers  who  wish  to  market  products  in  the 
United  States. 

•  Generalized  standards  are  identified  to  dis¬ 
tinguish  them  from  technical,  and  are  used 
when  specific  output  and  effectiveness  mea¬ 
sures  are  difficult  to  determine.  For  exam¬ 
ple,  certain  accrediting  organizations  devel¬ 
op  guidelines  and  standards  for  the  evalua¬ 
tion  and  certification  of  public  schools,  aca¬ 
demic  institutions,  and  libraries.  These 
standards  are  observed  and  maintained  by 
those  wishing  accreditation  (Teitz,  1968,  p. 

309)  . 

Unique  to  instructional  design  there  are  three  broad 
categories  of  instructional  effectiveness  measures:  (1)  stu¬ 
dent/group-oriented,  (2)  school-oriented;  and  (3)  community 
oriented . 


Student/Group  Oriented  Measures.  These  effectiveness 
measures  refer  to  academic  achievement  change  (grades)  result¬ 
ing  from  an  improvement  in  the  instructional  program. 


School -Ori ented  Measures.  These  effectiveness  measur¬ 
ers  are  characterized  by  three  types. 

•  The  change  in  number  dropping  out  of  school  as 
a  consequence  of  these  achievement  changes  in 
a  given  school  population  in  terms  of  changes . 

•  The  number  selecting  the  various  available 
courses  of  study. 

•  The  number  and  quality  of  those  graduating  (as 
indicated  by  scores  on  standardized  tests)  . 


Community-Oriented  Measures.  These  effectiveness  mea¬ 
sures  are  characterized  by  two  types. 

•  The  average  expected  lifetime  earnings  poten¬ 
tial  of  persons  with  different  levels  of  edu¬ 
cation  (economic  factors)  . 
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•  The  reduced  relationship  between  parental  socio¬ 
economic  level  and  individual  achievement  indi¬ 
cating  increased  equality  of  educational  oppor¬ 
tunity  (social  change)  (Abt  Associates,  Vol.  I, 

1967  ,  p.  3)  . 

Although  not  applicable  in  all  circumstances,  the  following 
ground  rules  may  be  observed  in  selecting  appropriate  measures 
of  effectiveness. 

•  Do  not  force  a  quantification  measure  of  effec¬ 
tiveness  where  no  sound  analytical  basis  for 

it  exists. 

•  Some  effectiveness  criteria  will  be  quantifi¬ 
able,  but  many  will  not  be. 

•  Nonquantif iable  effectiveness  criteria  pose 
no  particular  difficulty  in  analysis  when 
appropriate . 

•  When  the  analysis  involves  a  mixture  of  quan¬ 
titative  and  qualitative  effectiveness  criter¬ 
ia,  the  cost-effectiveness  analyst  must  exer¬ 
cise  caution  in  gauging  the  impact  of  the  qual¬ 
itative  criteria.  For  this  reason,  quantitative 
criteria  are  usually  preferred,  and  the  result 
is  interpreted  in  terms  of  the  qualitative  cri¬ 
teria  (Jakobsberg,  1966,  pp .  39-40). 

A  final  consideration  is  to  choose  a  measure  of  effec¬ 
tiveness  which  serves  as  a  sufficient  input  to  the  next  level 
of  decision-making  (Niskanen,  1969,  p.  30). 

A  methodological  consideration  discussed  in  Step  No.  4 
dealing  with  measures  of  both  cost  and  effectiveness  is  that 
no  single  criterion  measure  that  is  appropriate  for  all  goals/ 
objectives  in  known.  Even  if  a  single  criterion  measure  is 
adopted  for  all  goals/objectives,  the  present  inability  to 
quantify  relationships  suggests  the  use  of  a  multidimensional 
scale  for  measuring  results  (Packer,  1068,  p.  235). 


STEP  5:  DETERMINE  COST  OF  ALTERNATIVES 

Cost  and  effectiveness  of  alternatives  must  be  deter¬ 
mined  in  order  to  rationally  allocate  resources  .  Costs  are 
considered  easier  to  determine  than  effectiveness  because 
dollars,  manpower,  and  materials  are  homogeneous  and  measurable. 
Accurately  predicting  future  costs  is  not  simple,  however, 
because  consideration  must  be  given  to  uncertainty  of  cost 
estimates,  the  unpredictability  of  inflation,  and  to  the 


probable  time-value  of  money  due  to  discounting.  Other  fac¬ 
tors  that  could  also  be  included  are  the  cost  of  opportunities 
lost  and  the  effect  of  uncontrolled  consequences  of  the  out¬ 
comes  (Packer,  1968,  p.  247). 

The  problem  of  dollar-cost  estimating  often  assumes  so 
much  importance  in  any  cost-effectiveness  analysis  that  the 
usefulness  of  the  basic  concept  of  cost  (as  it  relates  to  the 
analysis)  is  lost.  As  a  result,  accounting/budget  costs  may 
take  on  an  importance  which  may  not  be  justified.  It  should 
be  remembered  that  costs  as  prediction  of  the  future  can  only 
be  estimates  and  we  always  estimate  future  values  on  the  basis 
of  past  ecperience.  As  a  consequence,  it  is  natural  that  re¬ 
cords  of  past  experience  are  reviewed  in  order  to  project  fu¬ 
ture  costs.  Since  the  accounting/budget  costs  records  usually 
include  some  aspects  of  actual  past  costs,  a  critical  review 
is  to  ensure  that  past  under  or  over  cost  expenditures  will 
not  prejudice  the  current  cost  estimate  (English,  1968,  p.  77) 

The  performance  of  Step  No.  5  should  deal  with  cost 
analysis  for  cost-effectiveness  and  similar  analysis.  The 
term  "cost  analysis,"  in  one  form  or  other,  is  also  used  for 
those  involved  in  budgeting,  accounting,  and  cost-control  pur¬ 
poses.  Although  somewhat  related,  these  methods  are  not  con¬ 
sidered  here  (Hatry,  1069,  p.  44).  In  cost-effectiveness 
analysis,  the  development  of  highly  detailed  cost  breakdowns 
are  not  required  (Novick,  1967,  p.  103).  It  should  be  also 
noted  that  the  preparation  of  the  cost  estimate  is  usually  per 
formed  by  someone  other  than  the  cost-effectiveness  analyst. 

It  is  axiomatic  that  the  estimate  cannot  be  better  than  the 
statement  of  the  goals/objectives/requirements  on  which  the 
cost  estimate  is  based.  These  statements,  however,  sometimes 
reflect  the  cost  estimator's  own  particular  biased  interests 
(Novick,  1963,  p.  102)  .  In  the  creation  of  a  cost  estimate 
in  accordance  with  the  goals/objectives/requirements,  the 
cost-effectiveness  analyst  must  verify  that  the  cost  estima¬ 
tor  has  provided  the  following  cost  data. 


Identify  Cost  Categories 

These  categories  cover  the  source  of  major  costs  (Herd, 
1965,  p.  81).  Measures  of  cost  (developed  in  Step  No.  4) 
should  be  indicated,  as  well  as  the  following  elements: 

•  Matter-energy,  information,  and  time. 

•  Human  effort  (student  time  and  effort,  teacher 
time  and  effort,  administrative  or  technician 
time  and  effort)  . 


•  Monetary  costs  (short  run  and  long  run  cap¬ 
ital  and  operating  activities)  (Miller  § 
Rath,  1969,  pp.  21-24)  . 


Identify  Resource  Constraints. 

These  constraints  usually  result  in  the  indication  that 
highly  aggregated  costs  are  useless;  that  costs  must  be  bro¬ 
ken  down  into  relatively  small  units  which  will  permit  a  vari¬ 
ety  of  cost  analysis  approaches. 


Calculate  Costs. 

Each  category  identified  should  be  considered  on  an 
appropriate  time  basis  as  well  as  in  total.  (Although  the 
alternative  total  cost  may  be  within  the  total  resources 
available,  the  annual  expenditures  may  not  be  compatible  with 
the  budget  or  funding  pattern.) 


Identify  Possible  Cost  Variances. 

A  single  cost  estimate  does  not  indicate  unknowns  re¬ 
lative  to  other  costs  provided  by  others.  Unknown  factors  in 
cost  data  elements  should  be  identified  for  subsequent  uncer¬ 
tainty  analysis. 


Verify  Completeness. 

Insure  compliance  of  cost  data  elements  with  a  check¬ 
list  (Herd,  1965,  p.  81)  . 

Essential  to  the  validity  of  all  cost-effectiveness  an¬ 
alysis  is  the  availability  of  appropriate,  accurate,  and  timely 
cost  estimates.  It  is  true  that  a  poor  cost-effectiveness  an¬ 
alysis  can  misuse  even  the  best  cost  data,  but  even  the  best 
of  analysis  cannot  do  much  with  inadequate  cost  data  (Hatry, 
1969,  p.  55) . 


STEP  6:  DETERMINE  CAPABILITIES  OR  EFFECTIVENESS  OF  ALTERNATIVES 

Once  the  effectiveness  evaluation  criteria  have  been 
determined  (Step  No.  4,  pp .  25-27,  above)  the  next  step  is  to 
express  the  capabilities  of  the  alternative  systems  in  terms 
of  the  evaluation  criteria.  To  save  time,  this  is  often  per¬ 
formed  concurrently  with  Step  No.  5.  Quantitative  expressions 
of  capabilities  are  preferred  when  they  are  available  but  qual¬ 
itative  expressions  are  acceptable  when  they  are  not  (Kazanowski, 
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1968,  p.  129).  As  is  cost  estimation,  the  estimates  of  the 
capabilities  of  alternative  systems  is  dependent  upon  infor¬ 
mation  and  data  supplied  by  someone  other  than  the  cost-effec¬ 
tiveness  analyst.  Although  this  is  both  an  advantage  as  well 
as  a  disadvantage,  confidence  in  the  accuracy  of  alternative 
system  capability  may  be  lacking  and  may  be  subject  to  the 
biased  interests  of  the  estimator  (Quade,  1969,  p.  10). 


STEP  7:  SELECT  FIXED-COST  OR  FIXED-EFFECTIVENESS  APPROACH 

In  assessing  alternatives,  the  procedure  may  now  take 
either  one  of  two  analytical  approaches: 

•  A  desired  level  of  effectiveness  may  be  speci¬ 
fied,  and  the  analyst  seeks  the  most  economi¬ 
cal  way  to  achieve  it,  or 

•  The  level  of  expenditure  may  be  specified  and 
the  analyst  explores  the  effectiveness  offered 
by  the  alternative  capabilities  (Breckner,  1967, 
p.  43)  . 

The  choice  of  specifying  either  the  fixed-cost  or  the 
fixed-effectiveness  analysis  approach  oftentimes  is  the  option 
of  the  analyst.  This  choice  is  important  because,  in  most 
cases,  real-world  problems  cannot  be  adequately  dealt  with  in 
such  a  simple  form  as  optimizing  either  cost  or  effectiveness. 
This  is  because  in  most  studies  unique  measures  of  cost  and 
measures  of  effectiveness  can  only  be  obtained  by  directing 
the  analysis  on  a  very  small  aspect  of  the  total  goals/objec¬ 
tives  and  requirements.  The  results  of  such  an  analysis  is 
rather  superficial  and  seldom  reflects  the  real-world  goals/ 
objectives  and  requirements  from  which  the  decision-maker  must 
select  the  best  alternative.  The  selection  of  one  approach, 
therefore,  may  more  closely  reflect  the  real-world  options 
actually  available  to  the  decision-maker,  whereas  the  selec¬ 
tion  of  another  approach  might  reflect  options  which  are  not 
available.  Another  factor  in  the  choice  of  one  approach  over 
the  other  may  also  yield  economies  of  analysis,  depending  on 
the  availability,  quality,  and  validity  of  data  to  be  utilized 
in  the  cost-effectiveness  analysis  itself.  The  choice  of 
fixed-cost  of  fixed-effectiveness  approach  is  least  affected 
by  the  constraints  imposed  by  the  decision-maker,  therefore, 
the  choice  of  approaches  is  vital  to  the  integrity  of  the  an¬ 
alysis. 


Fixed-Cost  Approach 

A  basic  fixed-cost  approach  is  first,  the  development 
of  alternative  systems  that  can  compete  in  meeting  the  goals/ 
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objectives/requirements  (Step  No.  3)  in  competition  for  the 
given  resources.  Second,  the  number  of  components  of  each 
alternative  that  can  be  developed,  procured,  and  operation¬ 
ally  implemented  within  the  fixed  resources,  is  determined. 
Finally,  the  alternative  which  satisfies  the  goals/objectives/ 
requirements  to  the  greatest  extent  is  identified  as  the  pre¬ 
ferred  choice. 


Fixed-Effectiveness  Approach 

A  basic  step  in  the  fixed-effectiveness  approach  is 
first,  development  of  a  measure  of  the  effectiveness  express¬ 
ed  in  terms  of  the  requirements  identified  in  the  prior  Step 
No.  2.  Then,  alternatives  are  developed  to  meet  the  goals/ 
objectives  as  in  Step  No.  3.  The  number  of  components  of 
each  alternative  necessary  to  attain  the  goals/objectives  is 
determined  and,  the  costs  and  penalties  incurred  by  each  al¬ 
ternative  are  estimated.  Finally,  the  alternative  that  ex¬ 
hibits  the  least  aggregate  cost  penalty  is  identified  as  the 
preferred  choice. 

It  is  suggested  that  if  inflexible  constraints  are  im¬ 
posed  on  the  resources  available,  the  use  of  the  fixed-cost 
approach  is  indicated.  Conversely,  if  inflexible  constraints 
are  imposed  on  the  effectiveness  required,  the  use  of  the  fixed¬ 
effectiveness  approach  is  indicated.  In  the  absence  of  other 
guidelines,  the  presence  of  both  a  large  number  of  significant 
measures  of  effectiveness  criteria  and  a  small  number  of  mea¬ 
sures  of  cost  criteria  usually  indicates  that  the  fixed-effec¬ 
tiveness  approach  should  be  used  (Kazanowski,  1968  ,  pp .  126- 
128)  . 


Note  that  in  using  either  the  fixed-cost  and  or  fixed¬ 
effectiveness  approaches  to  evaluate  alternatives,  the  mea¬ 
sure  of  criteria  (either  the  cost  criteria  or  the  effective¬ 
ness  criteria)  must  be  held  constant  so  as  to  have  a  common 
basis  for  comparison.  Methods  in  which  both  the  measures  of 
cost  and  the  measures  of  effectiveness  are  allowed  to  vary 
are  to  be  avoided  because  there  would  then  be  no  common  basis 
for  comparison  (Barfoot,  1963,  p.  3). 


STEP  8:  CREATE  ALTERNATIVE  VERSUS  CRITERIAN  ARRAY 

Ideally,  it  would  be  desirable  to  examine  the  cost  and 
effectiveness  evaluation  criteria  of  the  alternatives  in  the 
real  world  rather  than  in  the  simulated  world  of  cost-effec¬ 
tiveness  analysis.  For  many  obvious  reasons,  such  as  exces¬ 
sive  expense  and/or  the  unavailability  of  the  alternatives, 
this  is  not  possible  (Blumstein,  1969,  p.  33).  Two  different 


analytical  techniques  of  conducting  cost=ef f ectiveness  eval¬ 
uations  within  either  the  fixed-cost  or  fixed-effectiveness 
approach  are  often  encountered.  These  are:  (1)  the  tabular 
display  technique;  and,  (2)  the  model  technique. 


The  Tabular  Display  Technique 

The  tabular  display  technique  is  used  when  the  alter¬ 
natives  are  being  evaluated  by  measures  of  cost,  or  measures 
of  effectiveness,  or  by  both  measures  of  cost  and  effective¬ 
ness  that  are  incommensurate  with  one  another.  In  this  tech¬ 
nique,  the  evaluation  criteria  underlying  the  analysis  are 
identified  at  the  tops  of  columns  and  arranged  in  decreasing 
importance  of  criteria  from  left  to  right  as  indicated  in 
Figure  6.  The  alternatives  are  then  listed  vertically,  with 
the  alternative  that  meets  the  most  significant  criterion  to 
the  greatest  extent  listed  first,  and  so  on.  This  technique 
is  useful  when  numerous  alternatives  are  being  evaluated  be¬ 
cause  the  technique  can  be  used  to  eliminate  the  less  likely 
competitive  alternatives  and  allow  attention  to  be  focused  on 
the  two  or  three  major  competitors.  Whether  the  fixed-cost  or 
fixed-effectiveness  approach  is  used,  the  tabular  array  created 
will  be  very  similar.  In  the  fixed-cost  approach,  major  em¬ 
phasis  is  placed  on  the  effectiveness  attainable;  however, 
cost  data  indicating  how  the  total  cost  is  divided  and  phased 
is  significant  and  should  be  shown.  On  the  other  hand,  in  the 
fixed-effectiveness  approach,  major  emphasis  is  placed  on  the 
total  cost.  The  advantage  of  the  tabular  display  technique 
is  that  the  orderly  presentation  of  alternatives  cost  and  cap¬ 
ability  data  is  permitted  so  that  their  impact  on  the  evalua¬ 
tion  can  be  readily  discerned  and  discussed  along  with  the  sig¬ 
nificant  interrelationships.  Conclusions,  therefore,  can  be 
reached  by  visible  traceable  means. 


The  Model  Technique 

The  model  technique,  in  which  either  cost  models  or 
effectiveness  models  are  created,  is  usually  used  when  the 
basic  differences  between  the  alternative  systems  are  rela¬ 
tively  minor,  so  as  to  permit  the  valid  expression  of  their 
essential  differences  by  a  single  parameter  (Kazanowski,  1968, 
pp.  129,  134  §  136).  These  models  may  be  structured  along  a 
spectrum  of  abstraction  as  shown  in  Figure  7.  First,  and  the 
least  abstract  to  be  considered  is  the  real-world  system,  with¬ 
in  which  the  analyst  cannot  work.  Next  the  more  abstract  mo¬ 
dels  of  this  real-world  are  examined.  This  includes  the 
technique  of  operational  exercise,  gaming,  simulation,  and  an¬ 
alytical  modeling.  Last,  at  the  opposite  end  of  the  spectrum, 
is  total  abstraction,  within  which,  again,  the  analyst  cannot 
work . 
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Figure  6.  Tabular  Display  Technique  Example 
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Figure  7.  Model  Techniques  Spectrum 
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Operational  Exercise  Technique.  The  closest  approxi¬ 
mation  to  the  real-world  would  be  an  operational  exercise,  us¬ 
ing  the  actual  situations  with  the  alternatives  under  study. 
Operational  exercise  differs  from  the  real-world  in  that  the 
scale  is  smaller  and  many  of  the  personnel  are  simulated.  In 
addition,  the  consequences  of  failure  are  not  considered.  The 
operation  exercise,  consequently,  is  a  very  close  approxima¬ 
tion  to  the  real  alternative  but  contains  certain  factors  that 
would  not  normally  exist  in  the  real  world.  The  running  cost 
of  such  an  exercise  is  great  and  the  number  of  alternatives 
which  can  be  analyzed  is  small. 


Gaming  Technique.  The  next  region  down  this  spectrum 
of  abstraction  is  gaming.  Here  are  removed  from  the  represen¬ 
tation  of  the  real  world  those  components  that  can  most  easily 
be  simulated  by  a  simple  analog  such  as  a  computer  or  some 
other  device. 


Simulation  Technique.  The  next  stage  towards  model  ab¬ 
straction  is  to  physically  remove  the  human  decision-maker 
from  the  representation.  At  this  point  the  merits  and  opera¬ 
tion  of  the  alternatives  are  simulated  on  a  computer  in  which 
the  human  decision  rules  are  explicitly  programmed. 


Analytical  Technique.  In  those  models  already  consi¬ 
dered,  although  modified  or  compressed,  there  has  been  rela¬ 
tionships  between  time  in  the  real  world  and  time  in  the  model. 
In  analytical  models,  even  this  relationship  is  abandoned. 

Here  a  system  of  equations  is  created:  (1)  that  related  the 
characteristics  of  the  alternative;  and,  (2)  that  are  re¬ 
lated  to  the  measures  of  effectiveness.  Examples  of  this  are 
linear  programming  models,  graphic  models,  and  the  whole  class 
of  equations  referred  to  as  math  models. 


Model  Considerations 

The  model  spectrum  (Figure  7)  represents,  from  top  to 
bottom,  increasing  abstraction  from  the  real-world;  or  alter¬ 
nately,  proceeding  from  bottom  to  top  in  the  figure,  increas¬ 
ing  realism  or  closer  approximation  to  the  real  world.  It  is 
also  generally  observed  that  both  the  cost  of  the  analysis  and 
the  rate  at  which  we  can  examine  different  situations  decreas¬ 
es  as  the  model  used  becomes  more  abstract.  For  reasons  of 
economy  and  completeness  of  analysis,  we  would  like  to  operate 
as  close  to  total  abstraction  on  this  spectrum  as  possible. 

On  the  other  hand,  the  possibilities  of  faulty  assumptions  in¬ 
creases  as  we  move  away  from  working  in  the  real  world. 


In  exploring  considerations  that  go  into  modeling  tech¬ 
nique  selection  for  a  particular  cost-effectiveness  analysis 
it  should  be  remembered  that  the  basic  function  of  the  model 
is  to  determine  the  value  of  the  measure  of  effectiveness 
when  the  controllable  variables  of  the  alternatives  are  used 
(Blumstein,  1969,  pp .  34,  35  §  39).  Here  the  term  "model"  is 
used  in  a  broad  sense.  Depending  on  the  nature  of  the  goals/ 
objectives,  the  model  used  in  the  cost  or  effectiveness  anal¬ 
ysis  may  be  formal  or  informal,  very  mathematical  or  not  so 
mathematical,  heavily  computerized,  moderately  computerized, 
or  not  computerized  at  all.  The  main  point,  however,  is  that 
the  model  need  not  be  highly  formal  and/or  mathematical  to  be 
useful.  The  following  important  points  should  be  considered 
in  the  selection  of  modeling  techniques. 

•  Remember  that  model  building  is  an  art,  not  a 
science.  Often  it  is  an  experimental  process. 

•  Emphasize  those  factors  relevant  to  the  goals/ 
objectives,  and  suppress  those  factors  that 
are  relatively  unimportant.  The  model  is  like¬ 
ly  to  be  unmanageable  if  this  is  not  done. 

•  Develop  a  meaningful  set  of  relationships  among: 

(1)  goals/objectives,  (2)  the  alternatives 
available  for  attaining  the  goals/objectives, 

(3)  the  estimated  cost  of  the  alternatives; 
and,  (4)  the  estimated  effectiveness  for  each 
of  the  alternatives. 

•  Base  the  model  design  upon  the  "building-block" 
concept,  which  accepts  analytical  data  pre¬ 
pared  during  prior  steps  in  the  creation  of  a 
set  of  smaller  or  partial  models.  Each  small 
model  will  be  used  in  the  construction  of  a 
larger  subsequent  cost-effectiveness  model 
(Packer,  1968,  p.  236). 

•  Make  provisions  for  explicit  treatment  of  un¬ 
certainty  (see  Step  No.  10)  (Fisher,  1967,  p. 

72). 

Within  the  spectrum  of  modeling  technique  (from  concrete 
to  highly  abstract  models) ,  there  are  two  techniques  that  are 
most  appropriate  for  the  evaluation  of  measures  of  costs  and 
measures  of  effectiveness.  It  would  seem  that  the  choice 
should  be  dictated  by  the  category  of  the  cost-effectiveness 
study  as  well  as  the  goals/objectives.  This  choice  is  between 
simulation  and  analytical  modeling  techniques.  Although  the 
apparent  difference  between  these  two  techniques  is  sometimes 
small,  an  analytical  model  is  generally  more  efficient  if  the 
goals/objectives  can  be  stated  in  terms  such  that  available 
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mathematical  or  graphic  techniques  can  be  used  to  solve  for 
an  optimum  solution  (Packer,  1968,  p.  235).  Both  analysis  of 
cost  and  effectiveness  may  make  use  of  analytical  models  to 
represent:  (1)  each  alternatives  being  analyzed,  (2)  its 

operating  characteristics ;  and,  (3)  the  management  concepts 
of  its  operation  and  logistical  support.  These  models  are 
normally  structured  so  that  any  parameter,  or  combination  of 
parameters,  can  be  varied  to  determine  the  relative  effect  on 
the  alternatives'  total  cost  and  effectiveness  (Heaton,  1969, 
p.  35)  . 


There  are  several  considerations  which  should  be  con¬ 
sidered  in  the  utilization  of  analytical  models  for  both  cost 
and  effectiveness. 


Cost  Analytical  Mode)  Considerations.  The  cost  model 
relates  the  goals/objectives/requirements  and  activity  rates 
of  the  alternative,  and  of  the  personnel  that  operate  and/or 
support  it,  to  measures  of  cost  (Bryk,  1965,  p.  4).  Mathe¬ 
matical  cost  models  are  the  most  frequently  used  to  evaluate 
alternatives  when  goals/objectives  can  be  mathematically  for¬ 
mulated,  when  mathematical  conditions  can  be  met,  and  the  solu¬ 
tion  can  be  computed  (Miller  6  Rath,  1969,  p.  18).  The  key 
advantage  of  the  use  of  mathematical  cost  models  is  the  speed 
with  which  a  number  of  alternatives  can  be  costed.  While  ex¬ 
hibiting  significant  advantages  for  specific  applications, 
mathematical  cost  models  also  possess  substantial  limitations. 
Two  major  disadvantages  are: 

•  In  creating  the  cost  model  a  number  of  impli¬ 
cit  assumptions  are  made.  After  the  terms  of 
the  initial  equation  are  mathematically  mani¬ 
pulated  and  condensed  for  efficient  use,  the 
initial  elements  of  the  cost  are  no  longer 
visible.  Thus,  the  analyst  may  forget  his  in¬ 
itial  assumption  and  their  limitations. 

•  Mathematical  cost  models  are  relatively  in¬ 
flexible  and  good  basically  for  one  concept 
and  its  minor  variations.  If  a  basic  varia¬ 
tion  other  then  those  accounted  for  by  the 
model  is  to  be  costed,  the  model  must  be  re¬ 
vised  and  possibly  modified.  This  modifica¬ 
tion  must  be  verified  by  comparison  with  con¬ 
ventional  long-hand  results  and  as  a  conse¬ 
quence,  the  advantage  of  mathematical  cost 
modeling  is  lost  (Kazanowski,  1968,  p.  128). 

To  conserve  time,  cost  modeling  is  often  performed  concurrently 
with  effectiveness  modeling  (Bryk,  1965,  p.  14). 


Effectiveness  Analytical  Model  Considerations.  The 
effectiveness  model  relates:  (1)  the  measures  of  effective¬ 
ness  in  achieving  the  goals/objectives  to,  (2)  measures  of 
the  alternatives  operational  performance  (Bryk,  1965,  p.  2). 

The  use  of  a  mathematical  effectiveness  model  is  recommended 
when  the  alternatives  under  analysis  are  basically  so  similar 
that  those  evaluation  criteria  that  cannot  be  readily  vali¬ 
dated,  can  be  considered  to  cancel  each  other  out.  This  leaves 
only  the  quantifiable  and  commensurable  criteria  for  evalua¬ 
tion.  It  should  be  noted  that  this  occurrence  is  relatively 
infrequent  (Kazanowski,  1968,  p.  129). 


Analytical  Model  Computer  Use  Considerations.  Analytic 
cost-effectiveness  techniques  have  attained  their  present  level 
of  acceptance  largely  because  of  the  availability  of  today's 
high-speed  computers  and  the  development  of  appropriate  com¬ 
puter  models.  The  analytical  models  permit  the  study  of  com¬ 
plete  interactions  and  many  alternatives  heretofore  too  com¬ 
plex  and  time-consuming  for  manual  or  desk-calculator  analy¬ 
sis  (Bell,  1964,  p.2).  Prior  to  these  computer  programs,  the 
cost-effectiveness  analyst  had  to  choose  among  several  hand 
models,  which  were  often  unwieldy  and  unrealistic,  and  the 
older  prototype  computer  models.  These  older  computer  models 
were  frequently  inflexible  and  failed  to  give  insight  into 
how  conclusions  were  related  to  the  assumptions  (Quade,  1968, 
p.  243)  .  The  computerization  of  current  analytical  models  is 
warranted,  however,  only  when  the  analysis  of  alternatives  in¬ 
volves  repeated  computation  of  complex  functions  (or  \tfhen  the 
same  sub-routines  are  performed  repeatedly) .  In  addition, 
computerized  analytical  models  of  advanced  future  alternatives 
are  oftentimes  of  limited  value  because  the  expensive  program¬ 
ming  is  time-consuming  and  soon  obsolete  by  today's  standards. 

Concerning  cost  analytical  modeling,  if  the  model  is 
simple,  it  does  not  need  to  be  computerized:  if  it  is  com¬ 
plex,  the  maintenance  of  the  computer  model  and  associated 
documentation  may  become  time-consuming  and  expensive.  A  pos¬ 
sible  compromise  might  be  to  computerize  those  portions  of  an 
analytical  cost  model  that  are  not  subject  to  frequent  revi¬ 
sion.  In  a  study  directly  related  to  instructional  cost  model¬ 
ing,  it  was  indicated  that  complex  mathematical  modeling  would 
be  incompatible  with  the  concept  of  instructional  cost-effec¬ 
tiveness  development.  This  concept  was  to  develop  a  procedure 
that  was  easy  to  use  and  not  voluminous  (Institute  for  Educa¬ 
tional  Development,  1970,  pp .  3-4).  Regarding  effectiveness 
analytical  models,  these  are  seldom  mathematical  and  as  a  con¬ 
sequence,  computerization  is  seldom  applicable. 


Analytical  Model  Handbook  Considerations.  Rather  than 
computerize  generalized  all-purpose  analytical  models,  it  has 
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been  found  more  functional  to  compile  an  analytical  costing 
and  effectiveness  handbook.  Graphical  displays  of  cost  and 
effectiveness  relationships,  along  with  their  underlying 
goals/objectives/requirements  and  assumptions,  can  be  readily 
compiled  and  utilized.  The  handbook  both  allows  and  makes 
publicly  visible  the  exercise  of  judgment  to  ascribe  the  im¬ 
pact  of  subtle  deviations  from  a  basic  or  common  design  on 
the  cost  and  effectiveness  of  various  alternatives.  The  cost¬ 
ing/effectiveness  handbook  also  has  the  virtue  of  simplicity, 
speed,  flexibility,  and  economy  (Kazanowski,  1968,  pp .  131- 
132) .  The  Institute  for  Educational  Development  has  developed 
such  a  handbook,  for  alternative  instructional  design  selec¬ 
tion,  under  contract  for  the  Bureau  of  Naval  Personnel  and 
naval  training  schools  (Institute  for  Educational  Development, 
1970,  p.  1).  This  two-part  document  is  noteworthy  in  both 
volume  and  level  of  detail.  Another  approach  in  the  develop¬ 
ment  of  an  analytical  model  handbook  is  a  document  distribu¬ 
ted  by  the  Department  of  Defense  (Department  of  Defense,  1969, 
p.  1) .  This  both  outlines  the  procedure  of  analysis  and  en¬ 
closes  analytical  forms  to  be  used. 

It  should  be  emphasized  the  cost-effectiveness  models 
can  never  be  completely  realistic  for  they  are  dependent  on 
too  many  uncertain  parameters.  The  reliance  on  expert  judg¬ 
ment  is  indispensable  to  all  analysis.  Moreover,  the  virtue 
of  cost-effectiveness  analysis  is  the  provisions  of  a  frame¬ 
work  that  allows  the  judgment  and  intuition  of  specialists 
in  diverse  fields  to  be  combined.  This  framework  is  the  an¬ 
alytical  model  which  represents  a  simplified,  stylized  repre¬ 
sentation  of  those  aspects  of  the  real  world  as  appropriate 
to  the  goals/objectives  (Quade,  1968  ,  pp .  243,  246-247). 


STEP  9:  ANALYZE  MERITS  OF  ALTERNATIVES 

In  this  final  step,  the  integration  of  the  previously 
derived  alternative  cost  and  effectiveness  analytical  models 
is  performed.  The  purpose  of  the  integration  is  to  combine 
the  expected  values  of  alternative  cost  and  effectiveness  into 
a  single  common  framework  (Seiler,  1969,  p.  71).  This  frame¬ 
work  may  be  based  on  either  the  tabular  display  technique  or 
the  model  technique  discussed  in  Step  No.  8. 

An  example  of  the  tabular  display  technique  is  found  in 
Figure  8.  This  figure  is,  in  addition,  an  example  of  the  appli¬ 
cation  process  of  cost-effectiveness  analysis  (previously  dis¬ 
cussed)  in  which  the  analysis  is  both  structured  and  evalua¬ 
tion  is  completed.  This  display  was  developed  consistent  with 
information  provided  by  a  cost  and  effectiveness  input  simi¬ 
lar  to  that  found  in  Figure  6.  The  instructional  designs  un¬ 
der  study  represents  a  variety  of  alternatives  for  m-service 
employee  training.  These  alternatives  are  listed  in  order  of 
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RECOMMENDED  INSTRUCTIONAL  DESIGN  FOR  EMPLOYEE 
TRAINING  IN  ORDER  OF  PREFERENCE  ACCORDING  TO 
.  COST-EFFECTIVENESS  SELECTION  CRITERIA 


SELECTION  CRITERIA 

■ 

ALTERNATIVE 

INSTRUCTIONAL  DESIGN 

ECONOMY 

FLEXIBILITY 

ADMISSION  STANDARDS 

EMPLOYEE  ACCEPTABILITY 

EMPLOYER  ACCEPTABILITY 

SPEED 

TOTAL  NUMERICAL  EVAL. 

DAY-RELEASE  COURSES 

1  6  5  5  3  6 

26 

FULL  TIME  SCHOOL 

5  5  4  4  2  1 

21 

SANDWICH  COURSES 

4  4  3  3  4  3 

21 

VOCATIONAL  TRAINING 

6  16  6  12 

21 

BLOCK  RELEASE  COURSES 

3  3  2  2  6  4 

20 

EVENING  CLASSES 

man 

16 

Note:  After  Drovet,  1968,  p.  221 


Figure  8.  Cost-Effectiveness  Analysis 
Tabular  Display  Example 


41 


recommended  preference  according  to  the  cost-effectiveness 
selection  criteria  (Drouet,  1968,  p.  221)  on  an  evaluative 
scale  of  1  to  6  (6  equals  the  highest  or  most  preferred) . 

An  example  of  the  model  technique  is  presented  in  Fi¬ 
gure  9.  This  example  illustrates  both  a  graphic  analytical 
model  and  the  procedural  process  (also  previously  discussed) 
of  cost  effectiveness  analysis.  In  this  case,  the  analytical 
model  illustrates  the  logic  and  procedure  of  the  analysis  of 
each  alternative  instruction  design  for  service  training.  The 
model  indicates  that  each  alternative  instructional  design  may 
be  characterized  by:  (1)  instructional  requirements,  (2) 
instruction  time,  (3)  course  duration,  (4)  student  achieve¬ 
ment,  (5)  facility  requirements,  (6)  number  of  courses  per 
year,  (7)  equipment,  (8)  course  development,  (9)  instruc¬ 
tional  materials,  (10)  software  development,  and  (11)  stu¬ 
dent  and  instructor  travel  time.  In  addition,  each  instruc¬ 
tional  design  can  be  distinguished  by:  (1)  student  achieve¬ 
ment,  (2)  student  attrition  rate,  (3)  student  proficiency 
on  the  job  after  graduation,  and  (4)  student  morale.  Attri¬ 
butes  concerning  student  performance  and  morale,  during  in¬ 
struction  or  later,  are  identified  as  measures  of  effective¬ 
ness.  This  terminology  is  used  because  the  effectiveness  of 
the  instructional  design  is  measured  in  terms  of  the  end  pro¬ 
ducts;  i.e.,  student  course  completion.  Other  factors,  such 
as  instructor  time  per  course  and  amount  of  instructional  ma¬ 
terials  required  are  classified  as  measures  of  efficiency. 

The  analytical  model  indicates  that  feasibility  of  the  instruc 
tional  design  alternatives  are  determined  initially,  Next, 
the  characteristics  of  the  feasible  alternatives  are  listed 
as  well  as  the  differences  in  the  alternatives'  effectiveness, 
efficiency,  and  costs.  Risk  factors  are  also  considered  in 
the  analytical  model.  Last,  the  criterion  for  selecting  the 
instructional  design  is  applied  (Institute  for  Educational 
Development,  1969,  pp .  23,  25  §  26).  This  criterion  is  as 
follows : 

If  the  sum  of  incremental  benefits  in  dollars  of  the  new 
program  exceeds  its  incremental  benefits  in  dollars  of  the  new 
program  [sic.,  instructional  design] .. .exceeds  its  increase  in 
incremental  cost  [with  acceptable  risk] ,  then  the  new  program  is 
randed  higher  than  existing  program  and  the  existing  program  is 
eliminated  from  further  consideration.  If  the  reverse  is  true, 
the  existing  program  is  rated  higher  and  the  new  program  is  elim¬ 
inated  [Institute  for  Educational  Development,  1969,  p.  29], 


Regardless  of  the  technique  used,  both  the  cost-effec¬ 
tiveness  tabular  display  technique  and  the  analytical  model 
technique  relate  costs  and  effectiveness  so  that  the  merits  of 
the  alternatives  may  be  analyzed  for  the  selection  of  the  pre¬ 
ferred  alternative  (Bryk,  1965,  p.  4). 
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_ INPUTS 

INSTRUCTIONAL  OBJECTIVES 

•  NUMBER  AND  TYPES  OF  GRADUATES 

•  TYPES  OF  COURSES 
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•  SCHEDULE  FOR  PRODUCING  GRADUATES 
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TO  GRADUATE 
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•  INSTRUCTORS 
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TIVES  WITH 
RESOURCE 
REQUIRE¬ 
MENTS) 


ELIMINATE 
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ALTERNA¬ 
TIVES  FROM 
FURTHER 
CONSIDERA¬ 
TION 


Note:  After  Institute  for  Educational  Development, 
1969,  p.  42 


Figure  9.  Cost-Effectiveness  Analysis  Analytical 
Model  Example  (Sheet  1  of  5) 
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Figure  9.  Cost-Effectiveness  Analysis  Analytical 
Model  Example  (Sheet  2  of  5) 
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EFFECTIVENESS  MEASURERS 


From 

Shee 

No. 


INCREASES  IN  INSTRUC¬ 
TIONAL  EFFECTIVENESS 

OF  V  OVER  X 

I  REDUCTION  IN  STUDENT 
INSTRUCTION  TIME 

•  INCREASED  STUDENT 
EFFECTIVENESS  ON  THE 
JOB 

•  REDUCTION  IN  STUDENT 
ATTRITION  RATE 

•  IMPROVED  STUDENT 
MORALE 

•  EARLIER  AVAILABILITY 
FOR  WORK  ASSIGNMENT 

INCREASES  IN  INSTRUC¬ 
TIONAL  EFFICIENCY  OF 

Y  OVER  X 

•  REDUCTION  IN  INSTRUC¬ 
TOR  COURSE  TIME 

•  REDUCTION  IN  IN¬ 
STRUCTIONAL  MATER¬ 
IALS 

•  REDUCTION  IN  ADMIN¬ 
ISTRATION 

•  REDUCTION  IN  STUDENT 
TRAVEL  TIME  AND 
EXPENSES 

0  REDUCTION  IN  IN¬ 
STRUCTOR  TRAVEL  TIME 
AND  EXPENSES 


To 

Sheet 
No.  4 


Figure  9.  Cost-Effectiveness  Analysis  Analytical 
Model  Example  (Sheet  3  of  5) 


45 


From 
Sheet 
No.  3 


MEASURES 


From 
Sheet 
No.  2 


INCREASE  ON  INSTRUC¬ 
TIONAL  CAPITOL  COSTS 
OF  Y  OVER  X 

•  EQUIPMENT 

•  FACILITIES 

•  CONTENT  DEVELOPMENT 

•  SOFTWARE  DEVELOP¬ 
MENT 

•  INITIAL  SHAKEDOWN 

INCREASE  IN  INSTRUC¬ 
TIONAL  RECURRING 
COST  OF  Y  OVER  X 
«  RACILITY  RENTAL 

•  EQUIPMENT  RENTAL 

•  EQUIPMENT  MAINTE¬ 
NANCE 

•  INSTRUCTIONAL 
MATERIALS 


SUM 

INCREMENTAL 

BENEFITS 

To 

Sheet 

No .  5 

w 

t 

COST 

CONSIDER 

RISK 

FACTORS 

I 


SUM 

INCREMENTAL 
COSTS  OVER 
ALTERNA¬ 
TIVE'S 
LIFE 


To 

Sheet 
No .  5 


Figure  9.  Cost-Effectiveness  Analysis  Analytical 
Model  Example  (Sheet  4  of  5) 
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Figure  9.  Cost-Effectiveness  Analysis  Analytical 
Model  Example  (Sheet  5  of  5) 
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Once  the  alternatives  are  arranged  in  order  of  their 
acceptability  in  relationship  to  the  evaluation  criteria,  it 
is  generally  possible  to  eliminate  the  most  obviously  poorest 
competitors  and  focus  attention  on  the  top  three  or  four.  If 
the  effectiveness  and  cost  evaluation  outcomes  for  the  top  al¬ 
ternative  are  consistently  superior  to  the  respective  values 
of  the  other  alternatives,  then  that  alternative  is  the  pre¬ 
ferred  selection.  If  the  performance  values  for  the  top  al¬ 
ternatives  are  virtually  identical,  and  no  significant  differ¬ 
ence  in  cost  exists,  the  appropriate  conclusion  may  be  that 
there  is  no  significant  difference  between  the  top  candidates. 
In  this  case,  the  adoption  of  parallel  study  or  development 
efforts  of  both  top  candidates  may  be  indicated  in  order  to 
identify  the  preferred  alternative.  This  would  mean  that  one 
alternative  would  be  selected  at  a  later  date.  If  the  costs 
of  the  competing  alternatives  differ  significantly,  and  the 
evaluation  ratings  also  vary  significantly,  the  selection  may 
need  to  be  made  on  the  basis  of  personal  value  judgments 
(Kazanowski,  1968,  p.  135). 


STEP  10:  ANALYZE  UNCERTAINTY  FACTORS 

All  cost-effectiveness  analysis  studies  include  doubt¬ 
ful  features  that  may  not  be  satisfactorily  specified  or  quan¬ 
titatively  resolved  within  the  study  itself  (Brechner,  1967, 
p.  57) .  Important  decision  problems  involve  major  elements  of 
uncertainty.  Consequently,  a  cost-effectiveness  analysis  of 
such  problems  must  provide  for  the  explicit  treatment  of  un¬ 
certainty  for  consideration  by  the  decision-maker. 

Two  main  types  of  uncertainty  may  be  distinguished: 

(1)  uncertainty  about  the  state  of  the  \\rorld  in  the  future; 
and,  (2)  statistical  uncertainty  about  the  data  on  the  pre¬ 
sent  state  of  the  alternative.  State-of -the -future-uncertain¬ 
ty  stems  from  chance  elements  in  the  real  world  and  would 
exist  even  if  there  were  no  ties  of  the  state  to  the  world  of 
the  future  (Fisher,  1967,  pp.  72-73).  Statistical  data  un¬ 
certainty  is  most  closely  associated  with  cost-effectiveness 
analysis  because,  unfortunately,  the  choices  between  alterna¬ 
tive  candidates  are  seldom  made  on  the  basis  of  clear-cut  data. 
Factors  which  add  to  the  uncertainties  include:  (1)  alterna¬ 
tives  are  frequently  inadequate  to  fully  attain  the  objectives; 

(2)  measures  of  effectiveness  may  not  actually  measure  the 

extent  to  which  objectives  are  attained;  (3)  predictions  from 
the  cost-effectiveness  model  are  apt  to  be  full  of  uncertain¬ 
ties;  and,  (4)  other  criteria  which  look  almost  as  attrac¬ 
tive  as  the  criteria  chosen  may  lead  to  a  different  order  of 
preference  (Quade,  1969,  p.  5).  Three  types  of  uncertainty 
analysis  are  most  often  used  to  treat  statistical  uncertain¬ 
ties.  These  are  :  (1)  sensitivity  analysis;  (2)  contin¬ 

gency  analysis;  and,  (3)  a  Fortiori  analysis. 
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Sensitivity  Analysis 

In  many  instances  the  output  of  a  cost-effectiveness 
analysis  is  very  sensitive  to  the  assumptions  made.  The  con¬ 
clusions  reached  may  be  unknowingly  yet  significantly  biased 
by  some  apparently  innocuous  assumptions  which  are  essential 
to  the  analysis  (Kazanowski,  1968,  p.  138).  For  example, 
suppose  in  a  given  analysis  there  are  several  uncertain  key 
parameters.  In  stead  of  using  "expected  values"  for  these 
parameters,  the  analyst  could  test  for  the  influence  of  these 
assumptions  by  substituting  several  values  (i.e.,  high  and 
low)  in  an  attempt  to  see  how  sensitive  the  results  (the  rank¬ 
ing  of  the  alternatives  being  considered)  are  to  variations 
in  these  uncertain  parameters  (Fisher,  1967,  p.  73).  Another 
example  would  be  an  instance’  in  which  the  alternatives  are 
nearly  equal  in  merit.  Here,  it  \\rould  be  desirable  to  attempt 
to  estimate  the  cost-effectiveness  that  would  be  derived  from 
the  candidate  alternatives  if  they  were  cancelled  at  three  or 
four  different  major  life-cycle  schedule  milestones.  It  is 
also  desirable  to  perform  this  analysis  even  when  one  candi¬ 
date  alternative  appears  to  be  clearly  superior  with  respect 
to  both  cost  and  effectiveness  criteria.  Such  an  analysis  may 
point  out  the  need  for  caution  in  making  an  otherwise  unquali¬ 
fied  endorsement  of  the  alternative  (Kazanowski,  1968,  pp.  138 
139)  . 


Contingency  Analysis 

This  type  of  analysis  investigates  how  the  ranking  of 
the  candidate  alternatives  maintains  ranking  when  a  relevant 
change  in  criteria  for  evaluating  the  alternatives  is  postu¬ 
lated,  or  a  major  change  in  the  general  environment  is  intro¬ 
duced.  These  techniques  assess  the  degree  to  which  results 
are  contingent  upon  any  one  factor,  or  each  of  several  factors 


A  Fortiori  Analysis 

This  type  of  analysis  would  be  applicable  when  in  a 
planning  decision  problem  the  generally  accepted  intuitive 
judgment  strongly  favors  alternative  X.  The  analyst  feels, 
however,  that  X  might  be  a  poor  choice  and  that  alternative 
Y  might  actually  be  preferable.  In  performing  an  analysis  of 
X  versus  Y,  the  analyst  may  choose  deliberately  to  resolve  the 
major  uncertainties  in  favor  of  X  and  see  how  Y  compares  under 
adverse  conditions. 

Although  the  three  types  of  uncertainty  analysis  listed 
above  may  be  useful  in  a  direct  sense,  they  may  also  contri¬ 
bute  indirectly.  For  example,  through  sensitivity  and 
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contingency  analysis  the  analyst  may  improve  his  understanding 
of  critical  uncertainties  in  a  problem  area.  On  the  basis  of 
this  knowledge  he  might  then  be  able  to  create  a  new  alterna¬ 
tive  that  would  assume  more  protection  against  a  wider  range 
of  uncertainties.  This  is  often  difficult  to  do  but  when  it 
can  be  accomplished,  it  offers  one  of  the  best  ways  to  com¬ 
pensate  for  uncertainty  (Fisher,  1967,  pp .  73-74). 


STEP  11:  DOCUMENT  BASES  OF  PREVIOUS  STEPS  AND  SUBMIT  FINDINGS 

The  product  of  a  cost-effectiveness  analysis  study,  will 
in  all  probability  be  a  report  to  the  decision-maker.  In  sup¬ 
port  of  this  report  a  key  element  of  systematic  cost-effective¬ 
ness  analysis  is  sufficient  documentation  of  methods,  assump¬ 
tions,  sources,  etc.,  so  that  another  analyst  would  achieve 
substantially  the  same  results  with  the  same  material.  With¬ 
out  such  documented  results,  a  cost-effectiveness  analysis 
appeal  for  acceptance  rests  solely  on  faith  in  the  authority 
and  expertise  of  the  analyst  with  critical  examination  of  the 
way  in  which  he  arrived  at  the  recommendations  (Heymont,  1965 

p.  20)  . 


Particular  emphasis  should  be  placed  on  the  adequacy 
of  documenting  the  following: 

•  Specific  goals/objectives  to  be  attained. 

•  Essential  requirements  of  those  goals/objec¬ 
tives  along  with  associated  assumptions. 

•  Alternative  capabilities  and  associated  assump¬ 
tions  . 

•  Alternative  costs  and  associated  assumptions 
(learning  curves,  time,  quantities,  etc.). 

•  Alternative  evaluation  and  associated  assump¬ 
tions  (scenarios,  criteria,  etc.). 

•  Conclusions:  The  recommended  alternative,  the 
limitations,  and  the  associated  uncertainty 
factors . 

The  use  of  highly  esoteric  mathematics  for  documenta¬ 
tion  should  be  discouraged.  With  effort,  imagination,  and 
forethought,  the  analyst  can  usually  suitably  portray  complex 
mathematical  and  functional  relationships  in  simplified  and 
perhaps  graphic  form.  No  judicious  decision-maker  can  be  ex¬ 
pected  to  endorse  a  conclusion  or  recommendation  whose  ration¬ 
ale  and  derivation  he  cannot  fully  understand.  It  is  the 
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responsibility  of  the  analyst  to  present  the  documentation  in 
an  appropriate  and  understandable  manner.  To  have  a  high  pro¬ 
bability  of  acceptance  by  the  decision-maker  all  elements  of 
the  cost-effectiveness  analysis  must  be  documented  in  such  a 
manner  that  the  entire  process  can  be  clearly  followed  (Kaz- 
anowski ,  1968,  pp .  139-140). 


SUMMARY 


Cost-effectiveness  analysis  can  be  used  as  an  effective 
decision  assisting  aid: 

•  To  demonstrate  the  interactions  among  the  fac¬ 
tors  that  characterize  alternative  courses  of 
actions  . 

•  To  enable  the  consequences  of  these  alterna¬ 
tive  courses  to  be  systematically  and  objec¬ 
tively  assessed . 

•  To  promote  objective  decision-making  (Jakobs- 
berg,  1966 ,  p .  42)  . 


OPERATIONAL  FACTORS  OF  COST-EFFECTIVENESS  ANALYSIS 

Figure  10  was  developed  to  graphically  illustrate  the 
methods  and  operational  factors  available  for  cost-effective¬ 
ness  analysis.  Cost-effectiveness  analysis  is  a  methodology 
whereby  the  choice  of  one  of  three  broad  categories  of  stu¬ 
dies  may  be  performed.  These  are  (1)  system  configuration/ 
component  study;  (2)  system  comparison  study;  and,  (3) 
suprasystem  comparison  study.  The  next  involves  choice  of  the 
analytical  process  to  use.  This  choice  is  either:  (1)  the 
procedural  process;  or,  (2)  the  application  process.  The 
third  decision  involved  the  choice  of  elements  of  the  stand¬ 
ard  procedure  of  the  actual  cost-effectiveness  analysis  itself. 
This  was  a  choice  of  analytical  techniques  between:  (1)  the 
tabular  display  technique  and,  (2)  the  model  technique. 


FUNCTION  DIFFERENCES  AND  CHARACTERISTICS  OF 
COST-EFFECTIVENESS  ANALYSIS 

From  the  cost-effectiveness  analyst's  point  of  view,  the 
selection  of  study  category  and  analytical  process  to  be  used 
usually  is  determined  by  requirements  outside  of  the  control  of 
the  analyst.  Only  within  the  standard  procedure  of  the  cost- 
effectiveness  analysis  itself  does  the  analyst  have  a  choice  in 
analytical  technique.  Figure  11  was  developed  to  illustrate 
the  function,  differences,  and  characteristics  of  the  various 
cost-effectiveness  methods. 


i 
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Figure  10.  Cost-Effectiveness  Analysis 
Methodological  Elements 


i 
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COST-EFFECTIVENESS  ANALYSIS  STANDARD  PROCEDURE 


FUNCTION 


STEP  1 

REVIEW  DEFINITION 
OF  GENERAL  GOALS/ 
OBJECTIVES 


STEP  2 

REVIEW  IDENTIFICA¬ 
TION  OF  PARA¬ 
METERS  OR  REQUIRE¬ 
MENTS 


STEP  3 

REVIEW  DEVELOPMENT 
OF  ALTERNATIVES 


STEP  4 

ESTABLISH  COST 
AND  EFFECTIVENESS 
EVALUATION 
CRITERIA 


DIFFERENCES 


CHARACTERISTICS 


COST  MEASURES 
•  NON-TIME 
PHASED 
ANALYSIS 
t  TIME-PHASED 
ANALYSIS 


EFFECTIVENESS 

MEASURES 

•  PERFORMANCE 
IDENTIFIERS 

•  STANDARDS 

•  UNIQUE 
EDUCATIONAL 


CHOICE  DETERMINED 
BY  CATEGORY  OF 
STUDY 


CHOICE  DETERMINED 
BY  GOALS/OBJEC¬ 
TIVES/REQUIREMENTS 


STEP  5 

DETERMINE  COST 
OF  ALTERNATIVES 


Note:  After  Pearson,  1972,  pp.  112-114 


Figure  11.  Cost-Effectiveness  Analysis  Function,  Differences, 
and  Characteristics  (Sheet  1  of  3) 
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FUNCTION 

DIFFERENCES 

CHARACTERISTICS  I 

STEP  6 

DETERMINE  CAPABILI¬ 
TIES  OR  EFFECTIVE¬ 
NESS  OF  ALTERNA¬ 
TIVES 

STEP  7 

SELECT  FIXED-COST 

OR  FIXED-EFFEC¬ 
TIVENESS  APPROACH 

STEP  8 

CREATE  ALTERNA¬ 
TIVES  VERSUS 
CRITERION  ARRAY 

TABULAR  DISPLAY 
TECHNIQUE 

CHOICE  DETERMINED  1 
BY  EVALUATION  j 
CRITERIA;  1 
RECOMMENDED  WHEN  1 
CRITERIA  VARY  1 
WIDELY  | 

MODEL  TECHNIQUE 

•  OPERATIONAL 
EXERCISE 

•  GAMING 

•  SIMULATION 

NOT  RECOMMENDED  1 
FOR  INSTRUCTIONAL  1 
COST-EFFECTIVENESS  1 
ANALYSIS  BECAUSE  1 
OF  COMPLEXITY  1 

•  ANALYTICAL 

RECOMMENDED  FOR  | 
USE  WHEN  THE  GOALS/I 
OBJECTIVES/REQUIRE-I 
MENTS  CAN  BE  I 
STATED  IN  TERMS  1 
SUCH  THAT  MATHEMA-  1 
TICAL  AND  GRAPHIC  1 
TECHNIQUES  CAN  BE  1 
USED  1 

Figure  11.  Cost-Effectiveness  Analysis  Function,  Differences, 
and  Characteristics  (Sheet  2  of  3) 
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OF  ALTERNATIVES 


DIFFERENCES 


TABULAR  DISPLAY 
TECHNIQUE 


MODEL  TECHNIQUE 
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■  EXERCISE 
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WHEN  SELECTION  MAY 
BE  BASED  ON  INTUI¬ 
TIVE  JUDGMENT 
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PREVIOUS  STEPS 
AND  SUBMIT 
FINDINGS 


Figure  11.  Cost-Effectiveness  Analysis  Function,  Differences, 
and  Characteristics  (Sheet  3  of  3) 
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Function 

The  following  functional  elements  (in  the  form  of  a 
standard  step  procedure)  are  necessary  in  the  performance  of 
a  cost-effectiveness  analysis. 

•  Step  1:  Review  definition  of  general  goals/ 
objectives . 

•  Step  2:  Review  identification  of  parameters 
or  requirements . 

•  Step  3:  Review  development  of  alternatives. 

•  Step  4:  Establish  cost  and  effectiveness 
evaluation  criteria. 

•  Step  5:  Determine  cost  of  alternatives. 

•  Step  6:  Determine  capabilities  or  effective¬ 
ness  approach. 

•  Step  7:  Select  fixed-cost  or  f ixed- effective - 
ness  approach. 

•  Step  8:  Create  alternatives  versus  criterion 
array. 

•  Step  9:  Analyze  merits  of  alternatives. 

•  Step  10:  Analyze  uncertainty  factors. 

•  Step  11:  Document  basis  of  previous  steps  and 
submit  findings. 

The  grouping  of  the  above  functional  elements  inevitably  leads 
to  some  duplication  of  material,  but  without  the  performance 
of  these  elements,  the  cost-effectiveness  analysis  will  be  of 
lesser  quality  and  the  findings  questionable. 


Differences 

Differences  occur  in  the  performance  of  various  cost- 
effectiveness  analysis  in  several  functional  elements  of  the 
standard  procedural  steps.  These  differences  are  found  and 
described  in  some  detail  in  Step  4,  Step  8,  Step  9,  and  Step 
10  of  Figure  11. 
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Characteristics 

Characteristics  of  the  differences  in  Step  4,  Step  8, 
Step  9,  and  Step  10  of  the  functional  elements  of  the  stand¬ 
ard  procedure  are  also  considered  in  Figure  11.  It  was  noted, 
however,  that  the  principle  differences  in  cost-effectiveness 
analysis  methods  are  in  Step  8  (create  alternative  versus  cri¬ 
terion  array)  and  in  Step  9  (analyze  merits  of  alternatives) . 
These  two  differences  are  actually  the  same;  i.e.,  the  choice 
of  using  the  tabular  display  technique  or  the  analytical  model 
technique.  Thus,  this  is  the  only  methodological  choice  avail¬ 
able  to  the  cost-effectiveness  analyst  not  determined  by  others 
or  influenced  by  outside  conditions . 


CONCLUSION 

Cost-effectiveness  analysis,  in  contrast  to  other  manage¬ 
ment  aids,  provides  answers  by  a  systematic  process  which  is 
reproducible,  assessible  to  critical  examination,  and  readily 
modified  as  new  data  becomes  available.  At  the  very  least, 
cost-effectiveness  analysis  can  supply  a  means  for  selecting 
the  numerical  quantities  related  to  an  instructional  design 
in  a  logical  and  consistent  manner.  But  before  quantitative 
analysis  can  be  of  assistance  to  the  decision-maker,  it  must 
be  tempered  with  and  used  alongside  of  experience,  experimen¬ 
tation  and  judgment  (Quade,  1969,  p.  15).  Indeed,  cost-effec¬ 
tiveness  analysis  cannot  be  a  substitute  for  judgment.  Judg¬ 
ment  must  be  involved  in  determining  what  elements  shall  be 
considered  in  the  analysis  and  how  they  shall  enter.  There¬ 
fore,  cost-effectiveness  analysis  supplements  judgment  by  in¬ 
ductive  reasoning  which  is  fully  documented.  Human  judgment, 
however  well  informed,  can  still  lead  to  wrong  decisions.  It 
is  believed,  however,  that  cost-effectiveness  analysis  can  at 
least  enhance  the  possibility  that  the  decision  is  the  best 
one.  After  an  appropriate  analysis,  the  instructional  manage¬ 
ment  decision-maker  will  be  in  a  much  better  position  to  make 
a  decision- -although  he  may  not  find  the  decision  easier  to 
make  (Herd,  1965,  p.  79). 


APPENDIX 


DEFINITION  OF  TERMS  USED 

Cost.  Unit  of  resources  that  is  the  limiting  constraint. 
Dollars  are  used  in  most  cases  but  other  resources,  such  as 
manpower,  materials,  and  facilities  will  also  be  considered 
as  measurers  of  cost  (Hatry,  1969,  p.  44). 

Cost-benefit  analysis.  A  problem  solving  approach 
which  requires  the  definition  of  objectives,  and  identifica¬ 
tion  of  the  alternative  that:  (1)  yields  the  greatest  bene¬ 
fit  for  a  given  cost;  or  what  amounts  to  the  same  thing, 

(2)  that  yields  a  required  amount  of  benefits  for  the  least 
cost.  The  term  usually  applies  to  situations  in  which  the 
alternative  outputs  can  be  quantified  in  dollars.  A  chief 
characteristic  is  that  the  aim  is  to  calculate  the  present 
value  of  benefits  and  costs,  subject  to  specified  constraints. 

Cost-effectiveness  analysis.  A  problem  solving  approach 
which  requires  the  definition  of  objectives,  identification  of 
alternative  ways  of  achieving  the  objective,  and  identifica¬ 
tion  of  the  alternative  that:  (1)  yields  the  greatest  effec¬ 
tiveness  for  any  given  cost;  or  what  amounts  to  the  same 
thing,  (2)  that  yields  a  required  degree  of  effectiveness 
for  the  least  cost.  The  term  is  usually  used  in  applications 
in  which  the  alternative  outputs  cannot  be  easily  quantified 
in  dollars  (McGivney  and  Nelson,  1969,  p.  105). 

Cost-utility  analysis.  Same  as  cost-effectiveness  an¬ 
alysis  (Anshen,  1967,  p.  3).  Numerous  terms  currently  convey 
the  same  general  meaning  (i.e.,  systems  analysis,  operations 
analysis,  cost-effectiveness  analysis,  and  operations  research); 
however,  they  have  varying  connotations  to  different  people 
(Fisher,  1967,  p.  66).  Because  of  such  confusion  in  terminolo¬ 
gy  and  meaning,  cost-effectiveness  will  be  the  term  used  to  re¬ 
flect  the  level  of  intent  of  the  study. 

Discounted  cost.  An  analytical  technique  that  accounts 
for  the  fact  that  money  to  be  paid  in  the  future  yields  in¬ 
vestment  return  until  the  point  in  time  when  it  is  actually 
spent;  consequently,  present  money  spent  is  worth  less  than 
future  money  spent  (Seiler,  1969,  p.  17). 

Effective.  The  accomplishment  of  the  recognized  objec¬ 
tives  (Beynon,  1968,  p.  84). 
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Initial  cost.  Investment  in  goods  and  services  re¬ 
quired  to  establish  and  operate  a  system.  These  goods  have 
a  useful  value  of  longer  than  a  year;  thus  the  costs  are 
not  repeated  every  year  (Esseff ,  1970)  . 

Life  cycle  costs.  Combined  initial  costs  and  opera¬ 
tional  costs  for  the  estimated  useful  life  of  the  system 
(Esseff,  1970) . 

Model  .  Relations  used  to  portray  real  or  expected 
conditions,  actions,  or  effects  in  order  to  predict  the  out¬ 
come  of  actions  (Heymont,  1965,  p.  56). 

Objective.  A  statement  that  describes  in  observable 
and  measurable  terms  the  expected  output  performance  of  the 
product  of  the  system  (Banathy,  1968,  p.  89). 

Planning,  programming,  and  budgeting  system  (PPBS). 
Cost-benefit  analysis  on  a  large  scale  (Commission  on  Instruc¬ 
tional  Technology,  1070,  p.  90).  A  conceptual  approach  to 
decision-making  which  emphasizes  outputs,  program  activities, 
and  accomplishments.  Long-range  planning,  analytic  evalua¬ 
tive  tool's,  and  economic  rationality  are  basic  ingredients 
(Hartley,  1968,  p.  258). 

System.  An  integrated  relationship  of  people,  equip¬ 
ment,  and  methods  appropriately  organized  to  accomplish  de¬ 
fined  tasks  (Heymont,  1965,  p.  59). 

Systems  analysis  (systems  approach).  Self -correcting 
and  logical  methodology  of  decision-making  to  be  used  for  the 
design  development  of  manmade  entities.  Strategies  of  this 
methodology  include:  (1)  the  formulation  of  performance  ob¬ 
jective,  (2)  the  analysis  of  functions  and  components;  and, 
(3)  implementation  (Banathy,  1968,  p.  91). 

Time-phased  cost.  The  presentation  of  costs  by  the  time 
period  in  which  the  costs  occur  rather  than  a  total  cost 
figure  (Heymont,  1965,  p.  60). 

Trade-off.  The  weighing  of  alternative  means  to  be  em¬ 
ployed  for  the  accomplishment  of  required  functions  (Banathy, 
1968,  p.  91).  Cost-effectiveness  represents  one  aspect  of 
trade-off  analysis. 
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